Image-Based Methods for Interaction with Head-Worn Worker-Assistance Systems

In this paper, a mobile assistance-system is described which supports users in performing manual working tasks in the context of assembling complex products. The assistance system contains a head-worn display for the visualization of information relevant for the workflow as well as a video camera to acquire the scene. This paper is focused on the interaction of the user with this system and describes work in progress and initial results from an industrial application scenario. We present image-based methods for robust recognition of static and dynamic hand gestures in realtime. These methods are used for an intuitive interaction with the assistance-system. The segmentation of the hand based on color information builds the basis of feature extraction for static and dynamic gestures. For the static gestures, the activation of particular sensitive regions in the camera image by the user’s hand is used for interaction. An HMM classifier is used to extract dynamic gestures depending on motion parameters determined based on the optical flow in the camera image.

[1]  Mokhtar M. Hasan,et al.  Hand Gesture Modeling and Recognition using Geometric Features: A Review , 2012 .

[2]  Christopher M. Schlick,et al.  Design and evaluation of an augmented reality welding helmet , 2007 .

[3]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[4]  Willibald A. Günthner,et al.  Pick-by-Vision: A first stress test , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[5]  Rafiqul Zaman Khan,et al.  Hand Gesture Recognition: A Literature Review , 2012 .

[6]  Abdesselam Bouzerdoum,et al.  Skin segmentation using color pixel classification: analysis and comparison , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Thomas Alexander,et al.  Display Devices for Virtual Environments: Impact on Performance, Workload, and Simulator Sickness , 2008, IPT/EGVE.

[8]  Nassir Navab,et al.  Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[9]  Dieter Schmalstieg,et al.  Robust and unobtrusive marker tracking on mobile phones , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[10]  Frank Dürr,et al.  Efficient real-time trajectory tracking , 2011, The VLDB Journal.

[11]  Ayoub Al-Hamadi,et al.  A Novel System for Automatic Hand Gesture Spotting and Recognition in Stereo Color Image , 2009, J. WSCG.

[12]  P. Peer,et al.  Human skin color clustering for face detection , 2003, The IEEE Region 8 EUROCON 2003. Computer as a Tool..

[13]  Maurizio Gentilucci,et al.  On gesture and speech , 2015 .

[14]  Ayoub Al-Hamadi,et al.  Utilizing Invariant Descriptors for Finger Spelling American Sign Language Using SVM , 2010, ISVC.

[15]  Rafiqul Zaman Khan,et al.  Survey on Various Gesture Recognition Technologies and Techniques , 2012 .

[16]  Ramon Mas,et al.  Hand Tracking and Gesture Recognition for Human-Computer Interaction , 2005 .

[17]  Rüdiger Mecke,et al.  Distribution of visual attention in head-worn displays , 2008, APGV '08.

[18]  Kevin Curran,et al.  A new colour space for skin tone detection , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[19]  Andrew Zisserman,et al.  Hand detection using multiple proposals , 2011, BMVC.

[20]  Rüdiger Mecke,et al.  Mobile Augmented Reality in industrial applications: Approaches for solution of user-related issues , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[21]  Gunnar Farnebäck,et al.  Two-Frame Motion Estimation Based on Polynomial Expansion , 2003, SCIA.

[22]  A. Leroi‐Gourhan,et al.  Gesture and Speech , 1993 .

[23]  J. Bhattacharya,et al.  Hand Gesture Recognition for Sign Language : A New Hybrid Approach , 2011 .

[24]  Bernard Rimé,et al.  Fundamentals of nonverbal behavior , 1991 .

[25]  E.Y. Lam,et al.  Combining gray world and retinex theory for automatic white balance in digital photography , 2005, Proceedings of the Ninth International Symposium on Consumer Electronics, 2005. (ISCE 2005)..

[26]  James M. Rehg,et al.  Statistical Color Models with Application to Skin Detection , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[27]  F Dejong Estienne,et al.  [Voice and emotion]. , 1991, Revue de laryngologie - otologie - rhinologie.

[28]  Georgia Albuquerque,et al.  Tangible 3D: hand gesture interaction for immersive 3D modeling , 2005, EGVE'05.

[29]  Shamus P. Smith,et al.  Citation for Published Item: Use Policy Evaluating Distributed Cognitive Resources for Wayfinding in a Desktop Virtual Environment , 2022 .

[30]  Dieter Schmalstieg,et al.  Fast calibration for augmented reality , 1999, VRST '99.

[31]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[32]  Allan Hanbury,et al.  Color based skin classification , 2012, Pattern Recognit. Lett..

[33]  Deborah Hix,et al.  Active Text Drawing Styles for Outdoor Augmented Reality: A User-Based Study and Design Implications , 2007, 2007 IEEE Virtual Reality Conference.

[34]  Ayoub Al-Hamadi,et al.  Posture Recognition using Combined Statistical and Geometrical Feature Vectors based on SVM , 2009 .

[35]  Ayoub Al-Hamadi,et al.  Posture and Gesture Recognition for Human-Computer Interaction , 2009 .

[36]  Alex Waibel,et al.  Readings in speech recognition , 1990 .

[37]  F DejongEstienne,et al.  Voice and emotion , 1991 .

[38]  Heung-Il Suk,et al.  Hand gesture recognition based on dynamic Bayesian network framework , 2010, Pattern Recognit..

[39]  Bernd Fröhlich,et al.  The GlobeFish and the GlobeMouse: two new six degree of freedom input devices for graphics applications , 2006, CHI.