Stereo Vision-based Hand Gesture Recognition under 3D Environment

Abstract Human hand gestures as a natural way of communication with computers is becoming an emerging field of research due to its various applications like sign language recognition, human computer interaction, gaming, virtual reality etc. Hand gesture recognition under 2D environment has some limitations as the information about the other dimension (z-axis) is missed out. So, hand gesture recognition under 3D environment is becoming a growing field of research. In this paper, we have proposed a novel technique to detect hand gesture with forward and backward movement towards and away from the camera respectively. Our technique is based on stereo-vision and we have used a disparity map-based centroid movement and changing of its intensity as feature to recognize the gesture with conditional random field (CRF) as classifier. Stereo calibration and rectification is done to get the rectified images and correspondence gives the depth map. We tested our proposed method for Arabic numerals (0-9) and it worked efficiently with an average recognition rate of 88%.

[1]  Federico Tombari,et al.  Classification and evaluation of cost aggregation methods for stereo correspondence , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Ayoub Al-Hamadi,et al.  Improving Hand Gesture Recognition Using 3D Combined Features , 2009, 2009 Second International Conference on Machine Vision.

[3]  Ling Shao,et al.  Enhanced Computer Vision With Microsoft Kinect Sensor: A Review , 2013, IEEE Transactions on Cybernetics.

[4]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[5]  Agnès Just,et al.  Recognition of isolated complex mono- and bi-manual 3D hand gestures , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[6]  Xiaodong Yang,et al.  Histogram of 3D Facets: A characteristic descriptor for hand gesture recognition , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[7]  Kandarpa Kumar Sarma,et al.  A Conditional Random Field Based Indian Sign Language Recognition System under Complex Background , 2014, 2014 Fourth International Conference on Communication Systems and Network Technologies.

[8]  Hsiu-Chin Lin,et al.  Learning task constraints in operational space formulation , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[9]  David S. Sumam,et al.  Stereovision based 3D hand gesture recognition for pervasive computing applications , 2013, 2013 9th International Conference on Information, Communications & Signal Processing.

[10]  B. Cyganek.  An Introduction to 3D Computer Vision Techniques and Algorithms , 2009 .

[11]  Kurt Konolige,et al.  Projected texture stereo , 2010, 2010 IEEE International Conference on Robotics and Automation.

[12]  Stan Sclaroff,et al.  Sign Language Spotting with a Threshold Model Based on Conditional Random Fields , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Thomas B. Moeslund,et al.  Long-Term Occupancy Analysis Using Graph-Based Optimisation in Thermal Imagery , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[15]  Nasser Kehtarnavaz,et al.  Stereo matching via selective multiple windows , 2007, J. Electronic Imaging.

[16]  Vladimir Kolmogorov,et al.  Computing visual correspondence with occlusions using graph cuts , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[17]  Samir Otmane,et al.  A discrete Hidden Markov models recognition module for temporal series: Application to real-time 3D hand gestures , 2012, 2012 3rd International Conference on Image Processing Theory, Tools and Applications (IPTA).

[18]  Anupam Agrawal,et al.  Vision based hand gesture recognition for human computer interaction: a survey , 2012, Artificial Intelligence Review.

[19]  Abdesselam Bouzerdoum,et al.  Skin segmentation using color pixel classification: analysis and comparison , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.