A position and rotation invariant framework for sign language recognition (SLR) using Kinect

Sign language is the only means of communication for speech and hearing impaired people. Using machine translation, Sign Language Recognition (SLR) systems provide medium of communication between speech and hearing impaired and others who have difficulty in understanding such languages. However, most of the SLR systems require the signer to sign in front of the capturing device/sensor. Such systems fail to recognize some gestures when the relative position of the signer is changed or when the body occlusion occurs due to position variations. In this paper, we present a robust position invariant SLR framework. A depth-sensor device (Kinect) has been used to obtain the signer’s skeleton information. The framework is capable of recognizing occluded sign gestures and has been tested on a dataset of 2700 gestures. The recognition process has been performed using Hidden Markov Model (HMM) and the results show the efficiency of the proposed framework with an accuracy of 83.77% on occluded gestures.

[1]  Luc Van Gool,et al.  Real-time sign language letter and word recognition from depth data , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[2]  Bodo Rosenhahn,et al.  Real-Time Sign Language Recognition Using a Consumer Depth Camera , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[3]  Changsheng Xu,et al.  Discriminative Exemplar Coding for Sign Language Recognition With Kinect , 2013, IEEE Transactions on Cybernetics.

[4]  Hee-Deok Yang,et al.  Sign Language Recognition with the Kinect Sensor Based on Conditional Random Fields , 2014, Sensors.

[5]  Sergio Escalera,et al.  Multi-modal gesture recognition challenge 2013: dataset and results , 2013, ICMI '13.

[6]  Stan Sclaroff,et al.  Estimating 3D hand pose from a cluttered image , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[7]  Yi Li,et al.  Hand gesture recognition using Kinect , 2012, 2012 IEEE International Conference on Computer Science and Automation Engineering.

[8]  Xia Liu,et al.  Hand gesture recognition using depth data , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[9]  Samiul Monir,et al.  Rotation and scale invariant posture recognition using Microsoft Kinect skeletal tracking feature , 2012, 2012 12th International Conference on Intelligent Systems Design and Applications (ISDA).

[10]  Kongqiao Wang,et al.  A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[11]  Tinne Tuytelaars,et al.  Towards sign language recognition based on body parts relations , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[12]  Lale Akarun,et al.  Real time hand pose estimation using depth sensors , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[13]  Debi Prosad Dogra,et al.  Coupled HMM-based multi-sensor data fusion for sign language recognition , 2017, Pattern Recognit. Lett..

[14]  Alon Lerner,et al.  Enhanced interactive gaming by blending full-body tracking and gesture animation , 2010, SA '10.

[15]  E. Escobedo-Cardenas,et al.  A robust gesture recognition using hand local data and skeleton trajectory , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[16]  Richard Bowden,et al.  Sign Language Recognition , 2011, Visual Analysis of Humans.

[17]  Richard Bowden,et al.  Search-By-Example in Multilingual Sign Language Databases , 2011 .

[18]  Jake Araullo,et al.  The Leap Motion controller: a view on sign language , 2013, OZCHI.

[19]  Mauro Donadeo,et al.  Combining multiple depth-based descriptors for hand gesture recognition , 2014, Pattern Recognit. Lett..

[20]  Junsong Yuan,et al.  Robust hand gesture recognition with kinect sensor , 2011, ACM Multimedia.

[21]  Hanqing Lu,et al.  Fusing multi-modal features for gesture recognition , 2013, ICMI '13.

[22]  Mario Fernando Montenegro Campos,et al.  Real-Time Gesture Recognition from Depth Data through Key Poses Learning and Decision Forests , 2012, 2012 25th SIBGRAPI Conference on Graphics, Patterns and Images.

[23]  B. Watanapa,et al.  Human gesture recognition using Kinect camera , 2012, 2012 Ninth International Conference on Computer Science and Software Engineering (JCSSE).

[24]  Alan W. C. Tan,et al.  A feature covariance matrix with serial particle filter for isolated sign language recognition , 2016, Expert Syst. Appl..

[25]  Yuan Yao,et al.  Contour Model-Based Hand-Gesture Recognition Using the Kinect Sensor , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[26]  Eduardo Zalama Casanova,et al.  Hand Gesture Recognition for Deaf People Interfacing , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[27]  Pietro Zanuttigh,et al.  Hand gesture recognition with jointly calibrated Leap Motion and depth sensor , 2015, Multimedia Tools and Applications.

[28]  H. Hashimoto,et al.  Human motion tracking of mobile robot with Kinect 3D sensor , 2012, 2012 Proceedings of SICE Annual Conference (SICE).

[29]  Tinne Tuytelaars,et al.  Rank Pooling for Action Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Frederico G. Guimarães,et al.  Feature extraction in Brazilian Sign Language Recognition based on phonological structure and using RGB-D sensors , 2014, Expert Syst. Appl..

[31]  Raúl Rojas,et al.  Sign Language Recognition Using Kinect , 2012, ICAISC.

[32]  Debi Prosad Dogra,et al.  A multimodal framework for sensor based sign language recognition , 2017, Neurocomputing.

[33]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[34]  Partha Pratim Roy,et al.  A Novel framework of EEG-based user identification by analyzing music-listening behavior , 2017, Multimedia Tools and Applications.

[35]  Francisco Javier Díaz Pernas,et al.  A Kinect-based system for cognitive rehabilitation exercises monitoring , 2014, Comput. Methods Programs Biomed..

[36]  Nicolas Pugeault,et al.  Sign Language Recognition using Sequential Pattern Trees , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Houqiang Li,et al.  Sign Language Recognition using 3D convolutional neural networks , 2015, 2015 IEEE International Conference on Multimedia and Expo (ICME).

[39]  Thad Starner,et al.  American sign language recognition with the kinect , 2011, ICMI '11.

[40]  Debi Prosad Dogra,et al.  Study of Text Segmentation and Recognition Using Leap Motion Sensor , 2017, IEEE Sensors Journal.

[41]  Lale Akarun,et al.  Real Time Hand Pose Estimation Using Depth Sensors , 2013, Consumer Depth Cameras for Computer Vision.

[42]  Robin R. Murphy,et al.  Hand gesture recognition with depth images: A review , 2012, 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication.

[43]  Chafic Mokbel,et al.  Dynamic and Contextual Information in HMM Modeling for Handwritten Word Recognition , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  David W. Murray,et al.  Regression-based Hand Pose Estimation from Multiple Cameras , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[45]  Debi Prosad Dogra,et al.  A bio-signal based framework to secure mobile devices , 2017, J. Netw. Comput. Appl..

[46]  Debi Prosad Dogra,et al.  3D text segmentation and recognition using leap motion , 2017, Multimedia Tools and Applications.

[47]  Guang Li,et al.  Sign Language Recognition and Translation with Kinect , 2013 .