Overview of Human-Robot Collaboration in Manufacturing

Human-robot collaboration (HRC) in the manufacturing context aims to realise a shared workspace where humans can work side by side with robots in close proximity. In human-robot collaborative manufacturing, robots are required to adapt to human behaviours by dynamically changing their pre-planned tasks. However, the robots used today controlled by rigid native codes can no longer support effective human-robot collaboration. To address such challenges, programming-free and multimodal communication and control methods have been actively explored to facilitate the robust human-robot collaborative manufacturing. They can be applied as the solutions to the needs of the increased flexibility and adaptability, as well as higher effort on the conventional (re)programing of robots. These high-level multimodal commands include gesture and posture recognition, voice processing and sensorless haptic interaction for intuitive HRC in local and remote collaboration. Within the context, this paper presents an overview of HRC in manufacturing. Future research directions are also highlighted.

[1]  Kazuhiro Kosuge,et al.  Self-collision avoidance motion control for human robot cooperation system using RoBE , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[2]  Sotiris Makris,et al.  High Level Robot Programming Using Body and Hand Gestures , 2016 .

[3]  Lihui Wang,et al.  Energy-efficient trajectory planning for an industrial robot using a multi-objective optimisation approach , 2018 .

[4]  G. Hirzinger,et al.  Touch: The direct type of human interaction with a redundant service robot , 2001, Proceedings 10th IEEE International Workshop on Robot and Human Interactive Communication. ROMAN 2001 (Cat. No.01TH8591).

[5]  Hermann Ney,et al.  LSTM Neural Networks for Language Modeling , 2012, INTERSPEECH.

[6]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[7]  Alexander Zelinsky,et al.  Quantitative Safety Guarantees for Physical Human-Robot Interaction , 2003, Int. J. Robotics Res..

[8]  Antonio Bicchi,et al.  An atlas of physical human-robot interaction , 2008 .

[9]  Lihui Wang,et al.  Safety in Human-Robot Collaborative Assembly , 2018 .

[10]  Sotiris Makris,et al.  Seamless human robot collaborative assembly – An automotive case study , 2018, Mechatronics.

[11]  Dominik Henrich,et al.  Safe human-robot-cooperation: image-based collision detection for industrial robots , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[12]  Xiaohui Xie,et al.  Co-Occurrence Feature Learning for Skeleton Based Action Recognition Using Regularized Deep LSTM Networks , 2016, AAAI.

[13]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[14]  Aude Billard,et al.  Safety issues in human-robot interactions , 2013, 2013 IEEE International Conference on Robotics and Automation.

[15]  Lars Asplund,et al.  Intuitive industrial robot programming through incremental multimodal language and augmented reality , 2011, 2011 IEEE International Conference on Robotics and Automation.

[16]  Magdalena D. Bugajska,et al.  Building a Multimodal Human-Robot Interface , 2001, IEEE Intell. Syst..

[17]  Miao Li,et al.  The ‘Internet of Things’ enabled real-time scheduling for remanufacturing of automobile engines , 2018, Journal of Cleaner Production.

[18]  Angelika Peer,et al.  Evaluation of Technical Haptic Collaboration Partners in Terms of Human-Likeness and Task Performance , 2010 .

[19]  Reid G. Simmons,et al.  Sensor fusion for human safety in industrial workcells , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[20]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[21]  B. Nickolay,et al.  Image based 3D surveillance for flexible man-robot-cooperation , 2005 .

[22]  Tatsuo Arai,et al.  Human-Robot Collision Avoidance using a modified Social force Model with Body Pose and Face Orientation , 2013, Int. J. Humanoid Robotics.

[23]  Paulo Leitão,et al.  Agent-based distributed manufacturing control: A state-of-the-art survey , 2009, Eng. Appl. Artif. Intell..

[24]  Tullio Tolio,et al.  A probabilistic approach to workspace sharing for human–robot cooperation in assembly tasks , 2016 .

[25]  Csaba Kardos,et al.  Mixed-initiative assembly planning combining geometric reasoning and constrained optimization , 2018 .

[26]  Lihui Wang,et al.  From Intelligence Science to Intelligent Manufacturing , 2019, Engineering.

[27]  Lihui Wang,et al.  Cloud Robotics Towards a CPS Assembly System , 2018 .

[28]  Lihui Wang,et al.  Active Collision Avoidance for Human-Robot Collaborative Manufacturing , 2012 .

[29]  Hamed Saeidi,et al.  An integrated framework for human-robot collaborative assembly in hybrid manufacturing cells , 2016, 2016 IEEE International Conference on Automation Science and Engineering (CASE).

[30]  Vincent Padois,et al.  Energy Based Control for Safe Human-Robot Physical Interaction , 2016, ISER.

[31]  Sotiris Makris,et al.  On a human–robot workplace design and task allocation system , 2017, Int. J. Comput. Integr. Manuf..

[32]  Francisco A. Candelas,et al.  Safe human-robot interaction based on dynamic sphere-swept line bounding volumes , 2011 .

[33]  Lihui Wang,et al.  Current status and advancement of cyber-physical systems in manufacturing , 2015 .

[34]  Lihui Wang,et al.  Active collision avoidance for human–robot collaboration driven by vision sensors , 2017, Int. J. Comput. Integr. Manuf..

[35]  Holly A. Yanco,et al.  Classifying human-robot interaction: an updated taxonomy , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[36]  Alessandro De Luca,et al.  Sensorless Robot Collision Detection and Hybrid Force/Motion Control , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[37]  Duc Truong Pham,et al.  External Force Detection for Physical Human-Robot Interaction Using Dynamic Model Identification , 2017, ICIRA.

[38]  Philipp Koch,et al.  3D Multi-Sensor Data Fusion for Object Localization in Industrial Applications , 2014, ISR 2014.

[39]  Masatoshi Ishikawa,et al.  An Active Assistant Robotic System Based on High-Speed Vision and Haptic Feedback for Human-Robot Collaboration , 2018, IECON 2018 - 44th Annual Conference of the IEEE Industrial Electronics Society.

[40]  Alin Albu-Schäffer,et al.  Modular state-based behavior control for safe human-robot interaction: A lightweight control architecture for a lightweight robot , 2011, 2011 IEEE International Conference on Robotics and Automation.

[41]  Lihui Wang,et al.  Towards Robust Human-Robot Collaborative Manufacturing: Multimodal Fusion , 2018, IEEE Access.

[42]  Martin Buss,et al.  Human-Robot Collaboration: a Survey , 2008, Int. J. Humanoid Robotics.

[43]  Michael I. Jordan,et al.  Learning with Mixtures of Trees , 2001, J. Mach. Learn. Res..

[44]  Kyle B. Reed,et al.  Physical Collaboration of Human-Human and Human-Robot Teams , 2008, IEEE Transactions on Haptics.

[45]  Alois Knoll,et al.  Toward Efficient Robot Teach-in and Semantic Process Descriptions for Small Lot Sizes , 2015, RSS 2015.

[46]  Uwe Dombrowski,et al.  Simulation of human-robot collaboration by means of power and force limiting , 2018 .

[47]  Csaba Kardos,et al.  Decomposition approach to optimal feature-based assembly planning , 2017 .

[48]  Frank Weichert,et al.  Analysis of the Accuracy and Robustness of the Leap Motion Controller , 2013, Sensors.

[49]  Lihui Wang,et al.  Deep Learning-based Multimodal Control Interface for Human-Robot Collaboration , 2018 .

[50]  Lihui Wang,et al.  Innovative control of assembly systems and lines , 2017 .

[51]  Lihui Wang,et al.  Latest Advancement in CPS and IoT Applications , 2018 .

[52]  Andrew Y. C. Nee,et al.  Methodologies for immersive robot programming in an augmented reality environment , 2006, GRAPHITE '06.

[53]  Stephanie Rosenthal,et al.  An effective personal mobile robot agent through symbiotic human-robot interaction , 2010, AAMAS.

[54]  Jeffrey Too Chuan Tan,et al.  Triple stereo vision system for safety monitoring of human-robot collaboration in cellular manufacturing , 2011, 2011 IEEE International Symposium on Assembly and Manufacturing (ISAM).

[55]  Gerald Penn,et al.  Applying Convolutional Neural Networks concepts to hybrid NN-HMM model for speech recognition , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[56]  Alessandro De Luca,et al.  Integrated control for pHRI: Collision avoidance, detection, reaction and collaboration , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[57]  Vladimir Pavlovic,et al.  Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[58]  Trevor Darrell,et al.  DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition , 2013, ICML.

[59]  Hajime Asama,et al.  Smooth collision avoidance in human-robot coexisting environment , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[60]  Darwin G. Caldwell,et al.  Learning-based control strategy for safe human-robot interaction exploiting task and robot redundancies , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[61]  Oussama Khatib,et al.  A Depth Space Approach for Evaluating Distance to Objects , 2015, J. Intell. Robotic Syst..

[62]  Robert X. Gao,et al.  Symbiotic human-robot collaborative assembly , 2019, CIRP Annals.

[63]  Sandra Hirche,et al.  Control sharing in human-robot team interaction , 2017, Annu. Rev. Control..

[64]  Vera Hummel,et al.  Capability-based Task Allocation in Human-robot Collaboration , 2017 .

[65]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[66]  Toshiyuki Murakami,et al.  An approach to collision detection and recovery motion in industrial robot , 1989, 15th Annual Conference of IEEE Industrial Electronics Society.

[67]  Lihui Wang,et al.  Robotic Assembly Planning and Control with Enhanced Adaptability , 2012 .

[68]  Andrea Maria Zanchettin,et al.  Safety in human-robot collaborative manufacturing environments: Metrics and control , 2016, IEEE Transactions on Automation Science and Engineering.

[69]  R. Dillmann,et al.  Using gesture and speech control for commanding a robot assistant , 2002, Proceedings. 11th IEEE International Workshop on Robot and Human Interactive Communication.

[70]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[71]  Lihui Wang,et al.  Ubiquitous manufacturing system based on Cloud , 2017 .

[72]  Dominik Henrich,et al.  Multi-camera collision detection between known and unknown objects , 2008, 2008 Second ACM/IEEE International Conference on Distributed Smart Cameras.

[73]  Andrew Y. C. Nee,et al.  Robot programming using augmented reality: An interactive method for planning collision-free paths , 2009 .

[74]  Sotiris Makris,et al.  On a Human and Dual-arm Robot Task Planning Method , 2016 .

[75]  Kazuhiro Kosuge,et al.  Robot-human collaboration for new robotic applications , 1994, Proceedings of IECON'94 - 20th Annual Conference of IEEE Industrial Electronics.

[76]  Milad Geravand,et al.  Port-based modeling of human-robot collaboration towards safety-enhancing energy shaping control , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[77]  Lihui Wang,et al.  IoT-enabled Dynamic Optimisation for Sustainable Reverse Logistics , 2018 .

[78]  Saeid Nahavandi,et al.  Kinematic and dynamic modelling of UR5 manipulator , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[79]  Jörg Krüger,et al.  Spatial Programming for Industrial Robots through Task Demonstration , 2013 .

[80]  Aren Jansen,et al.  CNN architectures for large-scale audio classification , 2016, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[81]  Yan Song,et al.  Robust sound event recognition using convolutional neural networks , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[82]  László Monostori,et al.  Agent-based systems for manufacturing , 2006 .

[83]  Gary M. Bone,et al.  Real-time 3D Collision Avoidance Method for Safe Human and Robot Coexistence , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[84]  Lihui Wang,et al.  Remote human–robot collaboration: A cyber–physical system application for hazard manufacturing environment , 2020 .

[85]  Lihui Wang,et al.  Gesture recognition for human-robot collaboration: A review , 2017, International Journal of Industrial Ergonomics.

[86]  Dana Kulic,et al.  2016 International Symposium on Experimental Robotics , 2017, Springer Proceedings in Advanced Robotics.

[87]  Lihui Wang,et al.  Human-Robot Collaboration Demonstrator Combining Speech Recognition and Haptic Control , 2017 .

[88]  Feng Duan,et al.  Human-robot collaboration in cellular manufacturing: Design and development , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[89]  Robin R. Murphy,et al.  How UGVs physically fail in the field , 2005, IEEE Transactions on Robotics.

[90]  Lihui Wang,et al.  Current status of reconfigurable assembly systems , 2007, Int. J. Manuf. Res..

[91]  Matteo Parigi Polverini,et al.  Real-time collision avoidance in human-robot interaction based on kinetostatic safety field , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[92]  Antonio Bicchi,et al.  Integration of active and passive compliance control for safe human-robot coexistence , 2009, 2009 IEEE International Conference on Robotics and Automation.

[93]  Dominik Henrich,et al.  Sensor-Based Loops and Branches for Playback-Programmed Robot Systems , 2017, RAAD.

[94]  Alessandro De Luca,et al.  Exploiting robot redundancy in collision detection and reaction , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[95]  André Crosnier,et al.  Collaborative manufacturing with physical human–robot interaction , 2016 .

[96]  Jörg Krüger,et al.  Spatial programming for industrial robots based on gestures and Augmented Reality , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[97]  Yue Wang,et al.  Collaborative Assembly in Hybrid Manufacturing Cells: An Integrated Framework for Human–Robot Interaction , 2018, IEEE Transactions on Automation Science and Engineering.

[98]  Csaba Kardos,et al.  Context-dependent multimodal communication in human-robot collaboration , 2018 .

[99]  Yann LeCun,et al.  What is the best multi-stage architecture for object recognition? , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[100]  Alessandro De Luca,et al.  Robot Collisions: A Survey on Detection, Isolation, and Identification , 2017, IEEE Transactions on Robotics.

[101]  Alin Albu-Schäffer,et al.  Cartesian impedance control techniques for torque controlled light-weight robots , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[102]  John Norrish,et al.  Recent Progress on Programming Methods for Industrial Robots , 2010, ISR/ROBOTIK.

[103]  Shinya Morinaga,et al.  Collision Detection System for Manipulator Based on Adaptive Control Scheme , 2003 .

[104]  Lihui Wang,et al.  Deep learning-based human motion recognition for predictive context-aware human-robot collaboration , 2018 .

[105]  Terrence Fong,et al.  A Survey of Methods for Safe Human-Robot Interaction , 2017, Found. Trends Robotics.

[106]  Roland Menassa,et al.  Levels of human and robot collaboration for automotive manufacturing , 2012, PerMIS.

[107]  Nikolaos G. Tsagarakis,et al.  Safe human robot interaction via energy regulation control , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[108]  Klaus Bengler,et al.  Human Centered Assistance Applications for the working environment of the future , 2015 .

[109]  Sotiris Makris,et al.  Intuitive dual arm robot programming for assembly operations , 2014 .

[110]  Lihui Wang,et al.  Feature-based control and information framework for adaptive and distributed manufacturing in cyber physical systems , 2017 .

[111]  Lihui Wang,et al.  Combined strength of holons, agents and function blocks in cyber-physical systems , 2016 .

[112]  Lihui Wang,et al.  Depth camera based collision avoidance via active robot control , 2014 .

[113]  Yoji Yamada,et al.  Human-robot contact in the safeguarding space , 1997 .

[114]  Pedro Neto,et al.  High-level programming and control for industrial robotics: using a hand-held accelerometer-based input device for gesture and posture recognition , 2010, Ind. Robot.

[115]  Christoph Walter,et al.  A projection-based sensor system for safe physical human-robot collaboration , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[116]  Lihui Wang,et al.  A Virtual Training Based Programming-Free Automatic Assembly Approach for Future Industry , 2018, IEEE Access.

[117]  Lihui Wang,et al.  Vision-guided active collision avoidance for human-robot collaborations , 2013 .

[118]  Lihui Wang,et al.  Human-robot collaborative assembly in cyber-physical production: Classification framework and implementation , 2017 .

[119]  T. S. B. Sudarshan,et al.  Mobile Robot Programming by Demonstration , 2011, 2011 Fourth International Conference on Emerging Trends in Engineering & Technology.

[120]  Sotiris Makris,et al.  An approach for implementing power and force limiting in sensorless industrial robots , 2018 .

[121]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[122]  Alexander Verl,et al.  Cooperation of human and machines in assembly lines , 2009 .

[123]  Lihui Wang,et al.  Interface architecture design for minimum programming in human-robot collaboration , 2018 .

[124]  Sotiris Makris,et al.  Human–robot interaction review and challenges on task planning and programming , 2016, Int. J. Comput. Integr. Manuf..

[125]  Yoji Yamada,et al.  A failure-to-safety "Kyozon" system with simple contact detection and stop capabilities for safe human-autonomous robot coexistence , 1995, Proceedings of 1995 IEEE International Conference on Robotics and Automation.

[126]  Peter Korondi,et al.  Joint Level Collision Avoidance for Industrial Robots , 2012, SyRoCo.

[127]  Jens Peter Wulfsberg,et al.  Human Hybrid Robot: a new concept for supporting manual assembly tasks , 2013, Prod. Eng..

[128]  Sotiris Makris,et al.  An outlook on future assembly systems introducing robotic mobile dual arm workers , 2018 .

[129]  Jean Ponce,et al.  Learning mid-level features for recognition , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[130]  Dominik Henrich,et al.  3D Collision Detection for Industrial Robots and Unknown Obstacles using Multiple Depth Images , 2009 .

[131]  Lihui Wang,et al.  Classification, personalised safety framework and strategy for human-robot collaboration , 2018 .

[132]  Lihui Wang,et al.  Agent-based control system for next generation manufacturing , 1998, Proceedings of the 1998 IEEE International Symposium on Intelligent Control (ISIC) held jointly with IEEE International Symposium on Computational Intelligence in Robotics and Automation (CIRA) Intell.

[133]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[134]  Kristin Ytterstad Pettersen,et al.  Controlling Kuka Industrial Robots: Flexible Communication Interface JOpenShowVar , 2015, IEEE Robotics & Automation Magazine.

[135]  Juhan Nam,et al.  Multimodal Deep Learning , 2011, ICML.

[136]  Lorenzo Sabattini,et al.  Admittance control parameter adaptation for physical human-robot interaction , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[137]  Dragoljub Surdilovic,et al.  Compliance control with dual-arm humanoid robots: Design, planning and programming , 2010, 2010 10th IEEE-RAS International Conference on Humanoid Robots.

[138]  Martin Buss,et al.  Haptic Human–Robot Collaboration: Comparison of Robot Partner Implementations in Terms of Human-Likeness and Task Performance , 2011, PRESENCE: Teleoperators and Virtual Environments.

[139]  Guillaume Morel,et al.  Human force amplification with industrial robot : Study of dynamic limitations , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[140]  Weiming Shen,et al.  Agent-based distributed manufacturing process planning and scheduling: a state-of-the-art survey , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[141]  Lihui Wang,et al.  Robotic assembly planning and control with enhanced adaptability through function blocks , 2014, The International Journal of Advanced Manufacturing Technology.

[142]  André Crosnier,et al.  Multimodal control for human-robot cooperation , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[143]  Carlos Morato,et al.  Toward Safe Human Robot Collaboration by Using Multiple Kinects Based Real-Time Human Tracking , 2014, J. Comput. Inf. Sci. Eng..

[144]  Carme Torras,et al.  Learning Physical Collaborative Robot Behaviors From Human Demonstrations , 2016, IEEE Transactions on Robotics.

[145]  Soundar R. T. Kumara,et al.  Cyber-physical systems in manufacturing , 2016 .

[146]  Alessandro De Luca,et al.  Collision detection and reaction: A contribution to safe physical Human-Robot Interaction , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[147]  Wenjun Xu,et al.  Sensorless and adaptive admittance control of industrial robot in physical human−robot interaction , 2018, Robotics and Computer-Integrated Manufacturing.

[148]  James H. Graham A fuzzy logic approach for safety and collision avoidance in robotic systems , 1995 .

[149]  Christoph Walter,et al.  A projection-based sensor system for ensuring safety while grasping and transporting objects by an industrial robot , 2015, 2015 IEEE International Symposium on Robotics and Intelligent Sensors (IRIS).

[150]  Sotiris Makris,et al.  Design Considerations for Safe Human-robot Collaborative Workplaces , 2015 .