Improved Mutual Understanding for Human-Robot Collaboration: Combining Human-Aware Motion Planning with Haptic Feedback Devices for Communicating Planned Trajectory

In a collaborative scenario, the communication between humans and robots is a fundamental aspect to achieve good efficiency and ergonomics in the task execution. A lot of research has been made related to enabling a robot system to understand and predict human behaviour, allowing the robot to adapt its motion to avoid collisions with human workers. Assuming the production task has a high degree of variability, the robot’s movements can be difficult to predict, leading to a feeling of anxiety in the worker when the robot changes its trajectory and approaches since the worker has no information about the planned movement of the robot. Additionally, without information about the robot’s movement, the human worker cannot effectively plan own activity without forcing the robot to constantly replan its movement. We propose a novel approach to communicating the robot’s intentions to a human worker. The improvement to the collaboration is presented by introducing haptic feedback devices, whose task is to notify the human worker about the currently planned robot’s trajectory and changes in its status. In order to verify the effectiveness of the developed human-machine interface in the conditions of a shared collaborative workspace, a user study was designed and conducted among 16 participants, whose objective was to accurately recognise the goal position of the robot during its movement. Data collected during the experiment included both objective and subjective parameters. Statistically significant results of the experiment indicated that all the participants could improve their task completion time by over 45% and generally were more subjectively satisfied when completing the task with equipped haptic feedback devices. The results also suggest the usefulness of the developed notification system since it improved users’ awareness about the motion plan of the robot.

[1]  Sotiris Makris,et al.  Human–robot interaction review and challenges on task planning and programming , 2016, Int. J. Comput. Integr. Manuf..

[2]  Minna Lanz,et al.  AR-based interaction for human-robot collaborative manufacturing , 2020, Robotics Comput. Integr. Manuf..

[3]  Matthew O. Ward,et al.  Enhancing robot teleoperator situation awareness and performance using vibro-tactile and graphical feedback , 2011, 2011 IEEE Symposium on 3D User Interfaces (3DUI).

[4]  Domenico Prattichizzo,et al.  Cooperative human-robot haptic navigation , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[5]  Robert X. Gao,et al.  Symbiotic human-robot collaborative assembly , 2019, CIRP Annals.

[6]  Hossein Karimpour,et al.  A Mixed-Perception Approach for Safe Human–Robot Collaboration in Industrial Automation , 2020, Sensors.

[7]  Stefano Mauro,et al.  A Practical and Effective Layout for a Safe Human-Robot Collaborative Assembly Task , 2021, Applied Sciences.

[8]  Domenico Prattichizzo,et al.  Haptic wrist guidance using vibrations for Human-Robot teams , 2016, 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[9]  Andrea Maria Zanchettin,et al.  Operator Awareness in Human–Robot Collaboration Through Wearable Vibrotactile Feedback , 2018, IEEE Robotics and Automation Letters.

[10]  Hossein Karimpour,et al.  A Mixed-Perception Approach for Safe Human–Robot Collaboration in Industrial Automation , 2020, Sensors.

[11]  Przemyslaw A. Lasota,et al.  Analyzing the Effects of Human-Aware Motion Planning on Close-Proximity Human–Robot Collaboration , 2015, Hum. Factors.

[12]  Dmitry Berenson,et al.  Human-robot collaborative manipulation planning using early prediction of human motion , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[13]  Thomas B. Moeslund,et al.  Projecting robot intentions into human environments , 2016, 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[14]  Luzheng Bi,et al.  EMG-Based 3D Hand Motor Intention Prediction for Information Transfer from Human to Robot , 2021, Sensors.

[15]  Ramsundar Kalpagam Ganesan.  Mediating Human-Robot Collaboration through Mixed Reality Cues , 2017 .

[16]  Xiangjun Zou,et al.  Banana detection based on color and texture features in the natural environment , 2019, Comput. Electron. Agric..

[17]  Rui Li,et al.  Kiwifruit detection in field images using Faster R-CNN with VGG16 , 2019, IFAC-PapersOnLine.

[18]  Kuo-Hao Tang,et al.  Assessment of Handover Prediction Models in Estimation of Cycle Times for Manual Assembly Tasks in a Human–Robot Collaborative Environment , 2020 .

[19]  Petr Novák,et al.  HUMAN – ROBOT COLLABORATION IN INDUSTRY , 2016 .

[20]  Barbara Deml,et al.  Human-Aware Robotic Assistant for Collaborative Assembly: Integrating Human Motion Prediction With Planning in Time , 2018, IEEE Robotics and Automation Letters.

[21]  Rüdiger Dillmann,et al.  Anticipate your surroundings: Predictive collision detection between dynamic obstacles and planned robot trajectories on the GPU , 2015, 2015 European Conference on Mobile Robots (ECMR).

[22]  Chung Hyuk Park,et al.  Design of Effective Robotic Gaze-Based Social Cueing for Users in Task-Oriented Situations: How to Overcome In-Attentional Blindness? , 2020, Applied Sciences.

[23]  Rüdiger Dillmann,et al.  Transparent Robot Behavior by Adding Intuitive Visual and Acoustic Feedback to Motion Replanning , 2018, 2018 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[24]  Christopher D. Wickens,et al.  The Design and Evaluation of Visual and Tactile Warnings in Support of Space Teleoperation , 2012 .

[25]  Peter Korondi,et al.  Remote operation and assistance in human robot interactions with vibrotactile feedback , 2017, 2017 IEEE 26th International Symposium on Industrial Electronics (ISIE).

[26]  Dinesh Manocha,et al.  FCL: A general purpose library for collision and proximity queries , 2012, 2012 IEEE International Conference on Robotics and Automation.

[27]  Andrea Bonci,et al.  Human-Robot Perception in Industrial Environments: A Survey , 2021, Sensors.

[28]  Heather M. Ross,et al.  Better Teaming Through Visual Cues: How Projecting Imagery in a Workspace Can Improve Human-Robot Collaboration , 2018, IEEE Robotics & Automation Magazine.

[29]  Francesco Leali,et al.  Survey on human–robot collaboration in industrial settings: Safety, intuitive interfaces and applications , 2018, Mechatronics.

[30]  Francesco Chinello,et al.  Vibrotactile haptic feedback for human-robot interaction in leader-follower tasks , 2012, PETRA '12.

[31]  Pavel Smrz,et al.  Combining Interactive Spatial Augmented Reality with Head-Mounted Display for End-User Collaborative Robot Programming , 2019, 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN).

[32]  Wei Tech Ang,et al.  Hand and Object Segmentation from Depth Image using Fully Convolutional Network , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[33]  Xiangjun Zou,et al.  High-accuracy multi-camera reconstruction enhanced by adaptive point cloud correction algorithm , 2019, Optics and Lasers in Engineering.

[34]  Soh-Khim Ong,et al.  A novel augmented reality-based interface for robot path planning , 2014 .

[35]  Zhifeng Liu,et al.  Standing-Posture Recognition in Human–Robot Collaboration Based on Deep Learning and the Dempster–Shafer Evidence Theory , 2020, Sensors.

[36]  Maja J. Mataric,et al.  How Robot Verbal Feedback Can Improve Team Performance in Human-Robot Task Collaborations , 2015, 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[37]  Xiangjun Zou,et al.  Recognition and Localization Methods for Vision-Based Fruit Picking Robots: A Review , 2020, Frontiers in Plant Science.

[38]  Kyrre Glette,et al.  Multi 3D camera mapping for predictive and reflexive robot manipulator trajectory estimation , 2016, 2016 IEEE Symposium Series on Computational Intelligence (SSCI).