Context-Aware Human-Robot Collaborative Assembly

In human-robot collaborative manufacturing, industrial robots would work alongside the human workers who jointly perform the assigned tasks. Recent research work revealed that recognised human motions could be used as input for industrial robots control. However, the human-robot collaboration team still cannot work symbiotically. In response to the requirement, this chapter explores the potential of establishing context awareness between a human worker and an industrial robot for human-robot collaborative assembly. The context awareness between the human worker and the industrial robot is established by applying gesture recognition, human motion recognition and Augmented Reality (AR) based worker instruction technologies. Such a system works in a cyber-physical environment and is demonstrated by case studies.

[1]  Yun Fu,et al.  Prediction of Human Activity by Discovering Temporal Sequence Patterns , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[3]  Ling Shao,et al.  Enhanced Computer Vision With Microsoft Kinect Sensor: A Review , 2013, IEEE Transactions on Cybernetics.

[4]  Shai Avidan,et al.  Locally Orderless Tracking , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1997, EuroCOLT.

[6]  J. Geoffrey Chase,et al.  Human-Robot Collaboration: A Literature Review and Augmented Reality Approach in Design , 2008 .

[7]  Dorin Comaniciu,et al.  Real-time tracking of non-rigid objects using mean shift , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[8]  Shunzheng Yu,et al.  Hidden semi-Markov models , 2010, Artif. Intell..

[9]  Marti A. Hearst.  Trends & Controversies: Support Vector Machines , 1998, IEEE Intell. Syst..

[10]  Ming-Hsuan Yang,et al.  Incremental Learning for Robust Visual Tracking , 2008, International Journal of Computer Vision.

[11]  Joseph A. Paradiso,et al.  Electric Field Sensing For Graphical Interfaces , 1998, IEEE Computer Graphics and Applications.

[12]  Ming-Hsuan Yang,et al.  Visual tracking with online Multiple Instance Learning , 2009, CVPR.

[13]  Christopher D. Wickens,et al.  A model for types and levels of human interaction with automation , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[14]  Shwetak N. Patel,et al.  Whole-home gesture recognition using wireless signals , 2013, MobiCom.

[15]  Aaron F. Bobick,et al.  Parametric Hidden Markov Models for Gesture Recognition , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Václav Hlavác,et al.  Pose primitive based human action recognition in videos or still images , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Andrew W. Fitzgibbon,et al.  Accurate, Robust, and Flexible Real-time Hand Tracking , 2015, CHI.

[18]  Robert E. Schapire,et al.  The Boosting Approach to Machine Learning An Overview , 2003 .

[19]  Sundaram Suresh,et al.  Human action recognition using Meta-Cognitive Neuro-Fuzzy Inference System , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[20]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[21]  Simone Calderara,et al.  Visual Tracking: An Experimental Survey , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Tiancheng Li,et al.  Deterministic resampling: Unbiased sampling to avoid sample impoverishment in particle filters , 2012, Signal Process..

[23]  Lihui Wang,et al.  Gesture recognition for human-robot collaboration: A review , 2017, International Journal of Industrial Ergonomics.

[24]  Cataldo Guaragnella,et al.  A Neural Network Approach for Human Gesture Recognition with a Kinect Sensor , 2014, ICPRAM.

[25]  Julien Letessier,et al.  Visual tracking of bare fingers for interactive surfaces , 2004, UIST '04.

[26]  Rob Miller,et al.  3D Tracking via Body Radio Reflections , 2014, NSDI.

[27]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[28]  Robin R. Murphy,et al.  Hand gesture recognition with depth images: A review , 2012, 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication.

[29]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[30]  Fang Yuan,et al.  Static hand gesture recognition based on HOG characters and support vector machines , 2013, 2013 2nd International Symposium on Instrumentation and Measurement, Sensor Network and Automation (IMSNA).

[31]  Zoran Popovic,et al.  Articulated body deformation from range scan data , 2002, SIGGRAPH.

[32]  Deepak Ghimire,et al.  Geometric Feature-Based Facial Expression Recognition in Image Sequences Using Multi-Class AdaBoost and Support Vector Machines , 2013, Sensors.

[33]  Saeid Nahavandi,et al.  Teaching a Digital Performing Agent: Artificial Neural Network and Hidden Markov Model for recognising and performing dance movement , 2014, MOCO.

[34]  Robert X. Gao,et al.  Cloud-enabled prognosis for manufacturing , 2015 .

[35]  Cory D. Kidd,et al.  HUMANOID ROBOTS AS COOPERATIVE PARTNERS FOR PEOPLE , 2004 .

[36]  Masatoshi Ishikawa,et al.  High-speed Human / Robot Hand Interaction System , 2015, HRI.

[37]  Michael S. Ryoo,et al.  Human activity prediction: Early recognition of ongoing activities from streaming videos , 2011, 2011 International Conference on Computer Vision.

[38]  Ken Perlin,et al.  Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks , 2014, ACM Trans. Graph..

[39]  Jin Zhang,et al.  Learning hierarchical spatio-temporal pattern for human activity prediction , 2016, J. Vis. Commun. Image Represent..

[40]  Giuseppe Belgioioso,et al.  Home Automation Oriented Gesture Classification From Inertial Measurements , 2015, IEEE Transactions on Automation Science and Engineering.

[41]  Mircea Nicolescu,et al.  Vision-based hand pose estimation: A review , 2007, Comput. Vis. Image Underst..

[42]  Andrew W. Fitzgibbon,et al.  The Vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Ayoub Al-Hamadi,et al.  A Hidden Markov Model-based continuous gesture recognition system for hand motion trajectory , 2008, 2008 19th International Conference on Pattern Recognition.

[44]  Tae-Kyun Kim,et al.  Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  S. Abdul-Kareem,et al.  RETRACTED ARTICLE: Static hand gesture recognition using neural networks , 2014, Artificial Intelligence Review.

[46]  Matthew Turk,et al.  View-based interpretation of real-time optical flow for gesture recognition , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[47]  B. Watanapa,et al.  Human gesture recognition using Kinect camera , 2012, 2012 Ninth International Conference on Computer Science and Software Engineering (JCSSE).

[48]  Jiri Matas,et al.  P-N learning: Bootstrapping binary classifiers by structural constraints , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[49]  William T. Freeman,et al.  Bayesian Reconstruction of 3D Human Motion from Single-Camera Video , 1999, NIPS.

[50]  Gary R. Bradski,et al.  ORB: An efficient alternative to SIFT or SURF , 2011, 2011 International Conference on Computer Vision.

[51]  Nazo Haroon,et al.  Multiple Hand Gesture Recognition using Surface EMG Signals , 2016 .

[52]  Kyoung Mu Lee,et al.  Visual tracking via geometric particle filtering on the affine group with optimal importance functions , 2009, CVPR.

[53]  Frédo Durand,et al.  Capturing the human figure through a wall , 2015, ACM Trans. Graph..

[54]  Antonis A. Argyros,et al.  Efficient model-based 3D tracking of hand articulations using Kinect , 2011, BMVC.

[55]  Thad Starner,et al.  Visual Recognition of American Sign Language Using Hidden Markov Models. , 1995 .

[56]  Tarik Arici,et al.  Gesture Recognition using Skeleton Data with Weighted Dynamic Time Warping , 2013, VISAPP.

[57]  Luca Maria Gambardella,et al.  Max-pooling convolutional neural networks for vision-based hand gesture recognition , 2011, 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA).

[58]  L. Vygotsky.  Mind in Society: The Development of Higher Psychological Processes: Harvard University Press , 1978 .

[59]  Leif E. Peterson.  K-nearest neighbor , 2009, Scholarpedia.

[60]  Ahmad Said Tolba,et al.  An efficient algorithm for 3D hand gesture recognition using combined neural classifiers , 2012, Neural Computing and Applications.

[61]  Lihui Wang,et al.  Depth camera based collision avoidance via active robot control , 2014 .

[62]  S. Haykin.  Kalman Filtering and Neural Networks , 2001 .

[63]  Andrea Lockerd Thomaz,et al.  Tutelage and Collaboration for Humanoid Robots , 2004, Int. J. Humanoid Robotics.

[64]  Lihui Wang,et al.  Review: Advances in 3D data acquisition and processing for industrial applications , 2010 .

[65]  Rémi Ronfard,et al.  A survey of vision-based methods for action representation, segmentation and recognition , 2011, Comput. Vis. Image Underst..

[66]  David D. Denison,et al.  Nonlinear estimation and classification , 2003 .

[67]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[68]  Alexander Zelinsky,et al.  An algorithm for real-time stereo vision implementation of head pose and gaze direction measurement , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[69]  Subhendu Roy,et al.  Real-time Implementation of Electromyography for Hand Gesture Detection Using Micro Accelerometer , 2016 .

[70]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[71]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[72]  Fadel Adib,et al.  See through walls with WiFi! , 2013, SIGCOMM.

[73]  Vassilis Athitsos,et al.  Comparing gesture recognition accuracy using color and depth information , 2011, PETRA '11.

[74]  Yael Edan,et al.  Vision-based hand-gesture applications , 2011, Commun. ACM.

[75]  R. E. Kalman,et al.  A New Approach to Linear Filtering and Prediction Problems , 2002 .

[76]  Michael E. Tipping.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001, J. Mach. Learn. Res..

[77]  Rudolph van der Merwe,et al.  The unscented Kalman filter for nonlinear estimation , 2000, Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No.00EX373).

[78]  Yang Zhang,et al.  Tomo: Wearable, Low-Cost Electrical Impedance Tomography for Hand Gesture Recognition , 2015, UIST.

[79]  Shie-Jue Lee,et al.  A neuro-fuzzy approach for segmentation of human objects in image sequences , 2003, IEEE Trans. Syst. Man Cybern. Part B.

[80]  James J. Little,et al.  A Boosted Particle Filter: Multitarget Detection and Tracking , 2004, ECCV.

[81]  Jean-Christophe Nebel,et al.  Tracking Human Position and Lower Body Parts Using Kalman and Particle Filters Constrained by Human Biomechanics , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[82]  Yi Li,et al.  Hand gesture recognition using Kinect , 2012, 2012 IEEE International Conference on Computer Science and Automation Engineering.

[83]  Alexander Verl,et al.  Cooperation of human and machines in assembly lines , 2009 .

[84]  S. Mitra,et al.  Gesture Recognition: A Survey , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[85]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[86]  Eamonn J. Keogh,et al.  Derivative Dynamic Time Warping , 2001, SDM.

[87]  Bernhard Schölkopf,et al.  The Kernel Trick for Distances , 2000, NIPS.

[88]  Philip R. Cohen,et al.  Persistence, Intention, and Commitment , 2003 .

[89]  Tin Hninn Hninn Maung,et al.  Real-Time Hand Tracking and Gesture Recognition System Using Neural Networks , 2009 .

[90]  Juan M. Corchado,et al.  Fight sample degeneracy and impoverishment in particle filters: A review of intelligent approaches , 2013, Expert Syst. Appl..

[91]  Junseok Kwon,et al.  Tracking by Sampling Trackers , 2011, 2011 International Conference on Computer Vision.

[92]  Andrew Zisserman,et al.  Two-Stream Convolutional Networks for Action Recognition in Videos , 2014, NIPS.

[93]  Martin Buss,et al.  Human-Robot Collaboration: a Survey , 2008, Int. J. Humanoid Robotics.

[94]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.