Vision-based continuous sign language recognition using product HMM

This paper introduces a vision-based continuous sign language recognition (CSR) system. This CSR system can differentiate the signs in vocabulary and the non-signs. First, the continuous sign language is segmented into isolated sign segments. Then, the sign segment which can be interpreted by Product-HMMs (pHMM) is a sign, otherwise it is a non-sign. In the experiments, we test 40 signs from Taiwanese Sign Language. Our system achieves a good performance of sign recognition accuracy of 94.04%. We also test three continuous sign language which consist of 18∼23 different signs. The experimental results show that the average sign recognition recall rate is 74.5% and precision rate is 89%.

[1]  Yuntao Cui,et al.  A Learning-Based Prediction-and-Verification Segmentation Scheme for Hand Sign Image Sequence , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Surendra Ranganath,et al.  Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Stan Sclaroff,et al.  Spatiotemporal gesture segmentation , 2006 .

[4]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[5]  Petros Maragos,et al.  Product-HMMs for automatic sign language recognition , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[6]  Zeungnam Bien,et al.  Real-time recognition system of Korean sign language based on elementary components , 1997, Proceedings of 6th International Fuzzy Systems Conference.

[7]  Narendra Ahuja,et al.  Recognizing hand gesture using motion trajectories , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[8]  Marcel J. T. Reinders,et al.  Sign Language Recognition by Combining Statistical DTW and Independent Classification , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Wen Gao,et al.  Large-Vocabulary Continuous Sign Language Recognition Based on Transition-Movement Models , 2007, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[10]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[11]  Dimitris N. Metaxas,et al.  Handshapes and movements: Multiple-channel ASL recognition , 2004 .

[12]  Ming Ouhyoung,et al.  A real-time continuous gesture recognition system for sign language , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[13]  Karl-Friedrich Kraiss,et al.  Towards an Automatic Sign Language Recognition System Using Subunits , 2001, Gesture Workshop.

[14]  Alex Pentland,et al.  Active gesture recognition using partially observable Markov decision processes , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[15]  Dimitris N. Metaxas,et al.  A Framework for Recognizing the Simultaneous Aspects of American Sign Language , 2001, Comput. Vis. Image Underst..

[16]  Yasuo Horiuchi,et al.  Sign Language Recognition Based on Position and Movement Using Multi-Stream HMM , 2008, 2008 Second International Symposium on Universal Communication.

[17]  Daniel Kelly,et al.  Recognizing Spatiotemporal Gestures and Movement Epenthesis in Sign Language , 2009, 2009 13th International Machine Vision and Image Processing Conference.

[18]  Stan Sclaroff,et al.  Sign Language Spotting with a Threshold Model Based on Conditional Random Fields , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Salvatore Gaglio,et al.  A Framework for Sign Language Sentence Recognition by Commonsense Context , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[20]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Yuntao Cui,et al.  Appearance-Based Hand Sign Recognition from Intensity Image Sequences , 2000, Comput. Vis. Image Underst..