Hand Gesture Modeling and Recognition for Human and Robot Interactive Assembly Using Hidden Markov Models

Gesture recognition is essential for human and robot collaboration. Within an industrial hybrid assembly cell, the performance of such a system significantly affects the safety of human workers. This work presents an approach to recognizing hand gestures accurately during an assembly task while in collaboration with a robot co-worker. We have designed and developed a sensor system for measuring natural human-robot interactions. The position and rotation information of a human worker's hands and fingertips are tracked in 3D space while completing a task. A modified chain-code method is proposed to describe the motion trajectory of the measured hands and fingertips. The Hidden Markov Model (HMM) method is adopted to recognize patterns via data streams and identify workers' gesture patterns and assembly intentions. The effectiveness of the proposed system is verified by experimental results. The outcome demonstrates that the proposed system is able to automatically segment the data streams and recognize the gesture patterns thus represented with a reasonable accuracy ratio.

[1]  Ernesto Bribiesca,et al.  A new chain code , 1999, Pattern Recognit..

[2]  I. René J. A. te Boekhorst,et al.  Learning about natural human-robot interaction styles , 2006, Robotics Auton. Syst..

[3]  Michel Beaudouin-Lafon,et al.  Charade: remote control of objects using free-hand gestures , 1993, CACM.

[4]  Thomas B. Sheridan,et al.  Remote Manipulative Control with Transmission Delay , 1963 .

[5]  Yoji Yamada,et al.  Warning: to err is human [human-friendly robot dependability] , 2004, IEEE Robotics & Automation Magazine.

[6]  Jeffrey Too Chuan Tan,et al.  Cooperation Between a High-Power Robot and a Human by Functional Safety , 2011, J. Robotics Mechatronics.

[7]  Javaid Iqbal,et al.  On the Improvement of Multi-Legged Locomotion over Difficult Terrains Using a Balance Stabilization Method: , 2012 .

[8]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[9]  Alexandros André Chaaraoui,et al.  A review on vision techniques applied to Human Behaviour Analysis for Ambient-Assisted Living , 2012, Expert Syst. Appl..

[10]  J. Geoffrey Chase,et al.  Human Robot Collaboration: An Augmented Reality Approach—A Literature Review and Analysis , 2007 .

[11]  Yuqi Chen,et al.  2D Hand Tracking Based on Flocking with Obstacle Avoidance , 2014 .

[12]  Radu Bogdan Rusu,et al.  3D is here: Point Cloud Library (PCL) , 2011, 2011 IEEE International Conference on Robotics and Automation.

[13]  Dieter Fox,et al.  A large-scale hierarchical multi-view RGB-D object dataset , 2011, 2011 IEEE International Conference on Robotics and Automation.

[14]  Ajmal S. Mian,et al.  Using Kinect for face recognition under varying poses, expressions, illumination and disguise , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[15]  Toby Sharp,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR.

[16]  Ayman Habib,et al.  OpenSim: Open-Source Software to Create and Analyze Dynamic Simulations of Movement , 2007, IEEE Transactions on Biomedical Engineering.

[17]  David Zeltzer,et al.  A survey of glove-based input , 1994, IEEE Computer Graphics and Applications.

[18]  Pablo Gil,et al.  3D Visual Sensing of the Human Hand for the Remote Operation of a Robotic Hand , 2014 .

[19]  Dragoljub Surdilovic,et al.  PISA: Next Generation of Flexible Assembly Systems - From Initial Ideas to Industrial Prototypes , 2010, ISR/ROBOTIK.

[20]  Alois Knoll,et al.  Human workflow analysis using 3D occupancy grid hand tracking in a human-robot collaboration scenario , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[21]  Feng Duan,et al.  Application of the Assembly Skill Transfer System in an Actual Cellular Manufacturing System , 2012, IEEE Transactions on Automation Science and Engineering.

[22]  Bernadette Dorizzi,et al.  On Using the Viterbi Path Along With HMM Likelihood Information for Online Signature Verification , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[23]  Jian Huang,et al.  An assembly strategy scheduling method for human and robot coordinated cell manufacturing , 2011, Int. J. Intell. Comput. Cybern..

[24]  Ferdinando Cannella,et al.  Optimal Subtask Allocation for Human and Robot Collaboration Within Hybrid Assembly System , 2014, IEEE Transactions on Automation Science and Engineering.

[25]  Sonja Stork,et al.  Artificial Cognition in Production Systems , 2011, IEEE Transactions on Automation Science and Engineering.

[26]  Alois Knoll,et al.  A skill-based approach towards hybrid assembly , 2010, Adv. Eng. Informatics.

[27]  Andrew T. Irish,et al.  Trajectory Learning for Robot Programming by Demonstration Using Hidden Markov Model and Dynamic Time Warping , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[28]  Shozo Takata,et al.  Human and robot allocation method for hybrid assembly systems , 2011 .

[29]  L. Prasanth,et al.  HMM-Based Online Handwriting Recognition System for Telugu Symbols , 2007 .

[30]  Toshio Fukuda,et al.  Development of Production Robot System that can Assemble Products with Cable and Connector , 2011, J. Robotics Mechatronics.

[31]  Alexander Verl,et al.  Cooperation of human and machines in assembly lines , 2009 .

[32]  Dieter Fox,et al.  Sparse distance learning for object recognition combining RGB and depth information , 2011, 2011 IEEE International Conference on Robotics and Automation.

[33]  Jason M. Schwier,et al.  Behavior Detection Using Confidence Intervals of Hidden Markov Models , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[34]  Jochen Triesch,et al.  Robotic Gesture Recognition by Cue Combination , 1998, GI Jahrestagung.

[35]  Shonali Krishnaswamy,et al.  Mining data streams: a review , 2005, SGMD.

[36]  Robin R. Murphy,et al.  Review of Human Studies Methods in HRI and Recommendations , 2010, Int. J. Soc. Robotics.

[37]  Véronique Perdereau,et al.  Cooperative Tasks between Humans and Robots in Industrial Environments , 2012 .

[38]  Yael Edan,et al.  Vision-based hand-gesture applications , 2011, Commun. ACM.

[39]  Junji Yamato,et al.  Recognizing human action in time-sequential images using hidden Markov model , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[40]  Dieter Fox,et al.  RGB-D Mapping: Using Depth Cameras for Dense 3D Modeling of Indoor Environments , 2010, ISER.

[41]  Kerstin Dautenhahn,et al.  Methodology & Themes of Human-Robot Interaction: A Growing Research Field , 2007 .