A multimodal framework for sensor based sign language recognition

Abstract In this paper, we propose a novel multimodal framework for isolated Sign Language Recognition (SLR) using sensor devices. Microsoft Kinect and Leap motion sensors are used in our framework to capture finger and palm positions from two different views during gesture. One sensor (Leap Motion) is kept below the hand(s) while the other (Kinect) is placed in front of the signer for capturing horizontal and vertical movement of fingers during sign gestures. A set of features is next extracted from the raw data captured with both sensors. Recognition is performed separately by Hidden Markov Model (HMM) and Bidirectional Long Short-Term Memory Neural Network (BLSTM-NN) based sequential classifiers. In the next phase, results are combined to boost-up the recognition performance. The framework has been tested on a dataset of 7500 Indian Sign Language (ISL) gestures comprised with 50 different sign-words. Our dataset includes single as well as double handed gestures. It has been observed that, accuracies can be improved if data from both sensors are fused as compared to single sensor-based recognition. We have recorded improvements of 2.26% (single hand) and 0.91% (both hands) using HMM and 2.88% (single hand) and 1.67% (both hands) using BLSTM-NN classifiers. Overall accuracies of 97.85% and 94.55% have been recorded by combining HMM and BLSTM-NN for single hand and double handed signs.

[1]  Jake Araullo,et al.  The Leap Motion controller: a view on sign language , 2013, OZCHI.

[2]  Mauro Donadeo,et al.  Combining multiple depth-based descriptors for hand gesture recognition , 2014, Pattern Recognit. Lett..

[3]  Junsong Yuan,et al.  Robust hand gesture recognition with kinect sensor , 2011, ACM Multimedia.

[4]  Jovan Popović,et al.  Real-time hand-tracking with a color glove , 2009, SIGGRAPH 2009.

[5]  Yi Li,et al.  Hand gesture recognition using Kinect , 2012, 2012 IEEE International Conference on Computer Science and Automation Engineering.

[6]  Malin Premaratne,et al.  Hand gesture tracking and recognition system using Lucas-Kanade algorithms for control of consumer electronics , 2013, Neurocomputing.

[7]  Swapnil Belhe,et al.  Indian Sign Language Recognition Using Kinect Sensor , 2015, ICIAR.

[8]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..

[9]  Pietro Zanuttigh,et al.  Hand gesture recognition with leap motion and kinect devices , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[10]  Kongqiao Wang,et al.  A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[11]  Lale Akarun,et al.  Hand Pose Estimation and Hand Shape Classification Using Multi-layered Randomized Decision Forests , 2012, ECCV.

[12]  Tarik Arici,et al.  Gesture Recognition using Skeleton Data with Weighted Dynamic Time Warping , 2013, VISAPP.

[13]  Raúl Rojas,et al.  Sign Language Recognition Using Kinect , 2012, ICAISC.

[14]  Paul Geladi,et al.  Principal Component Analysis , 1987, Comprehensive Chemometrics.

[15]  Lale Akarun,et al.  Hierarchically constrained 3D hand pose estimation using regression forests from single frame depth data , 2014, Pattern Recognit. Lett..

[16]  Ayoub Al-Hamadi,et al.  Real-Time Capable System for Hand Gesture Recognition Using Hidden Markov Models in Stereo Color Image Sequences , 2008, J. WSCG.

[17]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Thad Starner,et al.  American sign language recognition with the kinect , 2011, ICMI '11.

[19]  M. F. Tolba,et al.  Arabic sign language recognition using leap motion sensor , 2014, 2014 9th International Conference on Computer Engineering & Systems (ICCES).

[20]  Gary R. Bradski,et al.  Stereo based gesture recognition invariant to 3D pose and lighting , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[21]  Faisal R. Al-Osaimi,et al.  Spatially Optimized Data-Level Fusion of Texture and Shape for Face Recognition , 2012, IEEE Transactions on Image Processing.

[22]  Igor Zubrycki,et al.  Using Integrated Vision Systems: Three Gears and Leap Motion, to Control a 3-finger Dexterous Gripper , 2014, Recent Advances in Automation, Robotics and Measuring Techniques.

[23]  Yuan Yao,et al.  Contour Model-Based Hand-Gesture Recognition Using the Kinect Sensor , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[24]  Janusz Konrad,et al.  A gesture-driven computer interface using Kinect , 2012, 2012 IEEE Southwest Symposium on Image Analysis and Interpretation.

[25]  Richa Singh,et al.  Leap signature recognition using HOOF and HOT features , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[26]  Frederico G. Guimarães,et al.  Feature extraction in Brazilian Sign Language Recognition based on phonological structure and using RGB-D sensors , 2014, Expert Syst. Appl..

[27]  Vassilis Athitsos,et al.  Comparing gesture recognition accuracy using color and depth information , 2011, PETRA '11.

[28]  Debi Prosad Dogra,et al.  Computer-Vision-Assisted Palm Rehabilitation With Supervised Learning , 2016, IEEE Transactions on Biomedical Engineering.

[29]  Jürgen Schmidhuber,et al.  Framewise phoneme classification with bidirectional LSTM and other neural network architectures , 2005, Neural Networks.

[30]  Yan Wen,et al.  A robust method of detecting hand gestures using depth sensors , 2012, 2012 IEEE International Workshop on Haptic Audio Visual Environments and Games (HAVE 2012) Proceedings.

[31]  Lei Li,et al.  Handwriting and Gestures in the Air, Recognizing on the Fly , 2013 .

[32]  Ankit Chaudhary,et al.  Tracking of Fingertips and Centers of Palm Using KINECT , 2011, 2011 Third International Conference on Computational Intelligence, Modelling & Simulation.

[33]  Kanad K. Biswas,et al.  Gesture recognition using Microsoft Kinect® , 2011, The 5th International Conference on Automation, Robotics and Applications.

[34]  Pietro Zanuttigh,et al.  Hand gesture recognition with jointly calibrated Leap Motion and depth sensor , 2015, Multimedia Tools and Applications.

[35]  Karl-Friedrich Kraiss,et al.  Robust Person-Independent Visual Sign Language Recognition , 2005, IbPRIA.

[36]  Mohamed Mohandes,et al.  Recognition of Two-Handed Arabic Signs Using the CyberGlove , 2013 .

[37]  Nathaniel Rossol,et al.  A Multisensor Technique for Gesture Recognition Through Intelligent Skeletal Pose Analysis , 2016, IEEE Transactions on Human-Machine Systems.

[38]  Chung-Lin Huang,et al.  Hand gesture recognition using a real-time tracking method and hidden Markov models , 2003, Image Vis. Comput..

[39]  Junsong Yuan,et al.  Robust Part-Based Hand Gesture Recognition Using Kinect Sensor , 2013, IEEE Transactions on Multimedia.

[40]  Ching-Hua Chuan,et al.  American Sign Language Recognition Using Leap Motion Sensor , 2014, 2014 13th International Conference on Machine Learning and Applications.

[41]  Riccardo Leonardi,et al.  XKin: an open source framework for hand pose and gesture recognition using kinect , 2014, The Visual Computer.

[42]  S. Abdul-Kareem,et al.  RETRACTED ARTICLE: Static hand gesture recognition using neural networks , 2014, Artificial Intelligence Review.

[43]  Ok-Hue Cho,et al.  A Study about Honey Bee Dance Serious Game for Kids Using Hand Gesture , 2014, MUE 2014.

[44]  Kazuhiko Yamamoto,et al.  Focus of attention for face and hand gesture recognition using multiple cameras , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[45]  Xilin Chen,et al.  Sparse Observation (SO) Alignment for Sign Language Recognition , 2016, Neurocomputing.

[46]  Mohamed F. Tolba,et al.  A Proposed Hybrid Sensor Architecture for Arabic Sign Language Recognition , 2014, IEEE Conf. on Intelligent Systems.

[47]  Carlos Sagüés,et al.  Human-Computer Interaction Based on Hand Gestures Using RGB-D Sensors , 2013, Sensors.

[48]  Soumik Mondal,et al.  Recognition of Isolated Indian Sign Language Gesture in Real Time , 2010, BAIP.

[49]  S. Majumder,et al.  Shape, texture and local movement hand gesture features for Indian Sign Language recognition , 2011, 3rd International Conference on Trendz in Information Sciences & Computing (TISC2011).

[50]  Debi Prosad Dogra,et al.  Segmentation and recognition of text written in 3D using Leap motion interface , 2015, 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR).

[51]  Khaled Assaleh,et al.  Glove-Based Continuous Arabic Sign Language Recognition in User-Dependent Mode , 2015, IEEE Transactions on Human-Machine Systems.

[52]  Bin Yu,et al.  Feature learning based on SAE-PCA network for human gesture recognition in RGBD images , 2015, Neurocomputing.

[53]  Cristina V. Lopes,et al.  Free-hand interaction with leap motion controller for stroke rehabilitation , 2014, CHI Extended Abstracts.

[54]  Khaled Assaleh,et al.  Low Complexity Classification System for Glove-Based Arabic Sign Language Recognition , 2012, ICONIP.

[55]  Nicolas Pugeault,et al.  Spelling it out: Real-time ASL fingerspelling recognition , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[56]  Ling Shao,et al.  Enhanced Computer Vision With Microsoft Kinect Sensor: A Review , 2013, IEEE Transactions on Cybernetics.

[57]  Robin R. Murphy,et al.  Hand gesture recognition with depth images: A review , 2012, 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication.

[58]  Weiqiang Wang,et al.  Recognition of In-air Handwritten Chinese Character Based on Leap Motion Controller , 2015, ICIG.

[59]  Chi-Tsun Cheng,et al.  Live demonstration: A HMM-based real-time sign language recognition system with multiple depth sensors , 2015, 2015 IEEE International Symposium on Circuits and Systems (ISCAS).