Multimodal Robot Programming Interface Based on RGB-D Perception and Neural Scene Understanding Modules

[1] R. Adamini, N. Antonini, A. Borboni, S. Medici, C. Nuzzi, R. Pagani, A. Pezzaioli, and C. Tonola. “User-friendly human-robot interaction based on voice commands and visual systems,” 2021 24th International Conference on Mechatronics Technology (ICMT), 2021, pp. 1–5. Search in Google Scholar

[2] A. Bochkovskiy, C. Wang, and H.M. Liao. “YOLOv4: Optimal speed and accuracy of object detection,” CoRR, vol. abs/2004.10934, 2020. Search in Google Scholar

[3] M. Chen, C. Liu, and G. Du. “A human–robot interface for mobile manipulator,” Intelligent Service Robotics, vol. 11, no. 3, 2018, pp. 269–278; doi: 10.1007/s11370-018-0251-3. Search in Google Scholar

[4] P. de la Puente, D. Fischinger, M. Bajones, D. Wolf, and M. Vincze. “Grasping objects from the floor in assistive robotics: Real world implications and lessons learned,” IEEE Access, vol. 7, 2019, pp. 123725–123735. Search in Google Scholar

[5] W. Dudek and T. Winiarski. “Scheduling of a robot’s tasks with the tasker framework,” IEEE Access, vol. 8, 2020, pp. 161449–161471. Search in Google Scholar

[6] V. Dutta, and T. Zielińska. “Predicting human actions taking into account object affordances,” Journal of Intelligent & Robotic Systems, vol. 93, 2019, pp. 745–761. Search in Google Scholar

[7] V. Dutta, and T. Zielińska. “Prognosing human activity using actions forecast and structured database,” IEEE Access, vol. 8, 2020, pp. 6098–6116. Search in Google Scholar

[8] Q. Gao, J. Liu, Z. Ju, and X. Zhang. “Dual-hand detection for human–robot interaction by a parallel network based on hand detection and body pose estimation,” IEEE Transactions on Industrial Electronics, vol. 66, no. 12, 2019, pp. 9663–9672. Search in Google Scholar

[9] M. Gualtieri, J. Kuczynski, A.M. Shultz, A. Ten Pas, R. Platt, and H. Yanco. “Open world assistive grasping using laser selection,” 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017, pp. 4052–4057. Search in Google Scholar

[10] K. He, G. Gkioxari, P. Dollár, and R. Girshick. “Mask R-CNN,” 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2980–2988. Search in Google Scholar

[11] B. Kulecki. “Intuitive robot programming and interaction using RGB-D perception and CNN-based objects detection,” Automation 2022: New Solutions and Technologies for Automation, Robotics and Measurement Techniques, R. Szewczyk, C. Zieliński, and M. Kaliczyńska, eds., Cham, 2022, pp. 233–243. Search in Google Scholar

[12] B. Kulecki, and D. Belter. “Intuicyjny interfejs programowania robota z neuronowymi modułami do interpretacji sceny,” Postępy robotyki. T.2, C. Zieliński and A. Mazur, eds., Warszawa, 2022, pp. 89–98. Search in Google Scholar

[13] B. Kulecki, and D. Belter. “Robot programming interface with a neural scene understanding system,” Proceedings of the 3rd Polish Conference on Artifiicial Intelligence, April 25–27, 2022, Gdynia, Poland, 2022, pp. 130–133. Search in Google Scholar

[14] B. Kulecki, K. Młodzikowski, R. Staszak, and D. Belter. “Practical aspects of detection and grasping objects by a mobile manipulating robot,” Industrial Robot, vol. 48, no. 5, 2021, pp. 688–699. Search in Google Scholar

[15] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C.L. Zitnick. “Microsoft COCO: Common objects in context,” Computer Vision – ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, eds., Cham, 2014, pp. 740–755. Search in Google Scholar

[16] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A.C. Berg. “SSD: Single shot multibox detector,” Computer Vision – ECCV 2016, B. Leibe, J. Matas, N. Sebe, and M. Welling, eds., Cham, 2016, pp. 21–37. Search in Google Scholar

[17] X. Lv, M. Zhang, and H. Li. “Robot control based on voice command,” 2008 IEEE International Conference on Automation and Logistics, 2008, pp. 2490–2494. Search in Google Scholar

[18] I. Maurtua, I. Fernández, A. Tellaeche, J. Kildal, L. Susperregi, A. Ibarguren, and B. Sierra. “Natural multimodal communication for human–robot collaboration,” International Journal of Advanced Robotic Systems, vol. 14, no. 4, 2017; doi: 10.1177/1729881417716043. Search in Google Scholar

[19] O. Mazhar, S. Ramdani, B. Navarro, R. Passama, and A. Cherubini. “Towards real-time physical human-robot interaction using skeleton information and hand gestures,” IEEE/RSJ Int. Conf. on Int. Robots and Systems (IROS), 2018, pp. 1–6. Search in Google Scholar

[20] C. Nuzzi, S. Pasinetti, M. Lancini, F. Docchio, and G. Sansoni. “Deep learning-based hand gesture recognition for collaborative robots,” IEEE Instrumentation Measurement Magazine, vol. 22, no. 2, 2019, pp. 44–51. Search in Google Scholar

[21] K.-B. Park, S. H. Choi, J. Y. Lee, Y. Ghasemi, M. Mohammed, and H. Jeong. “Hands-free human–robot interaction using multimodal gestures and deep learning in wearable mixed reality,” IEEE Access, vol. 9, 2021, pp. 55448–55464. Search in Google Scholar

[22] P. Putthapipat, C. Woralert, and P. Sirinimnuankul. “Speech recognition gateway for home automation on open platform,” 2018 International Conference on Electronics, Information, and Communication (ICEIC), 2018, pp. 1–4. Search in Google Scholar

[23] S. Sharan, T. Q. Nguyen, P. Nauth, and R. Araujo. “Implementation and testing of voice control in a mobile robot for navigation,” 2019 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), 2019, pp. 145–150. Search in Google Scholar

[24] M. Simão, P. Neto, and O. Gibaru. “Natural control of an industrial robot using hand gesture recognition with neural networks,” IECON 2016 – 42nd Annual Conference of the IEEE Industrial Electronics Society, 2016, pp. 5322–5327. Search in Google Scholar

[25] R. Staszak, B. Kulecki, W. Sempruch, and D. Belter. “What’s on the other side? A single-view 3D scene reconstruction,” 2022 17th International Conference on Control, Automation, Robotics and Vision (ICARCV), 2022, pp. 173–180. Search in Google Scholar

[26] F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann. “Mediapipe hands: On-device real-time hand tracking,” arXiv e-prints, Jun 2020; doi: 10.48550/arXiv.2006.10214. Search in Google Scholar