A Real-Time ASL Recognition System Using Leap Motion Sensors

It is always challenging for deaf and speech-impaired people to communicate with non-sign language users. A real-time sign language recognition system using 3D motion sensors could lower the aforementioned communication barrier. However, most existing gesture recognition systems are adopting a single sensor framework, whose performance is susceptible to occlusions. In this paper, we proposed a real-time multi-sensor recognition system for American sign language (ASL). Data collected from Leap Motion sensors are fused using multiple sensors data fusion (MSDF) and the recognition is performed using hidden Markov models (HMM). Experimental results demonstrate that the proposed system can deliver higher recognition accuracy over single-sensor systems. Due to its low implementation cost and higher accuracy, the proposed system can be widely deployed and bring conveniences to sign language users.

[1]  Lale Akarun,et al.  Real Time Hand Pose Estimation Using Depth Sensors , 2013, Consumer Depth Cameras for Computer Vision.

[2]  Richard M. Schwartz,et al.  On-line cursive handwriting recognition using speech recognition methods , 1994, Proceedings of ICASSP '94. IEEE International Conference on Acoustics, Speech and Signal Processing.

[3]  Dimitris N. Metaxas,et al.  Parallel hidden Markov models for American sign language recognition , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[4]  Pramod K. Varshney,et al.  Multisensor Data Fusion , 1997, IEA/AIE.

[5]  Horst Bunke,et al.  Off-line cursive handwriting recognition using hidden markov models , 1995, Pattern Recognit..

[6]  K. Dill,et al.  Using quaternions to calculate RMSD , 2004, J. Comput. Chem..

[7]  Xiaodong Yang,et al.  Histogram of 3D Facets: A characteristic descriptor for hand gesture recognition , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[8]  Chi-Tsun Cheng,et al.  Live demonstration: A HMM-based real-time sign language recognition system with multiple depth sensors , 2015, 2015 IEEE International Symposium on Circuits and Systems (ISCAS).

[9]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[10]  Biing-Hwang Juang,et al.  Hidden Markov Models for Speech Recognition , 1991 .

[11]  Andrea Fossati,et al.  Consumer Depth Cameras for Computer Vision , 2013, Advances in Computer Vision and Pattern Recognition.

[12]  W. Kabsch.  A solution for the best rotation to relate two sets of vectors , 1976 .

[13]  Junsong Yuan,et al.  Robust hand gesture recognition based on finger-earth mover's distance with a commodity depth camera , 2011, ACM Multimedia.

[14]  LiXiaolin,et al.  Training Hidden Markov Models with Multiple Observations-A Combinatorial Method , 2000 .

[15]  Raman K. Mehra,et al.  Covariance intersection algorithm for distributed spacecraft state estimation , 2001, Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148).

[16]  James Llinas,et al.  Multisensor Data Fusion , 1990 .

[17]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[18]  Jin-Hyung Kim,et al.  An HMM-Based Threshold Model Approach for Gesture Recognition , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Pascal Vasseur,et al.  Introduction to Multisensor Data Fusion , 2005, The Industrial Information Technology Handbook.