Adapting hidden Markov models for ASL recognition by using three-dimensional computer vision methods

We present an approach to continuous American sign language (ASL) recognition, which uses as input 3D data of arm motions. We use computer vision methods for 3D object shape and motion parameter extraction and an ascension technologies 'Flock of Birds' interchangeably to obtain accurate 3D movement parameters of ASL sentences, selected from a 53-sign vocabulary and a widely varied sentence structure. These parameters are used as features for hidden Markov models (HMMs). To address coarticulation effects and improve our recognition results, we experimented with two different approaches. The first consists of training context-dependent HMMs and is inspired by speech recognition systems. The second consists of modeling transient movements between signs and is inspired by the characteristics of ASL phonology. Our experiments verified that the second approach yields better recognition results.

[1]  Scott K. Liddell,et al.  American Sign Language: The Phonological Base , 2013 .

[2]  Kuldip K. Paliwal,et al.  Automatic Speech and Speaker Recognition: Advanced Topics , 1999 .

[3]  Alex Pentland,et al.  Real-time American Sign Language recognition from video using hidden Markov models , 1995 .

[4]  Ioannis A. Kakadiaris,et al.  Active part-decomposition, shape and motion estimation of articulated objects: a physics-based approach , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Mohammed Waleed Kadous,et al.  Machine Recognition of Auslan Signs Using PowerGloves: Towards Large-Lexicon Recognition of Sign Lan , 1996 .

[6]  Annelies Braffort.  ARGo: An Architecture for Sign Language Recognition and Interpretation , 1996, Gesture Workshop.

[7]  KwangYun Wohn,et al.  Recognition of space-time hand-gestures using hidden Markov model , 1996, VRST.

[8]  Pavel Laskov,et al.  A MULTI-STAGE APPROACH TO FINGERSPELLING AND GESTURE RECOGNITION , 1996 .

[9]  Dimitris N. Metaxas.  Physics-Based Deformable Models: Applications to Computer Vision, Graphics, and Medical Imaging , 1996 .

[10]  M. B. Waldron,et al.  Isolated ASL sign recognition system for deaf persons , 1995 .

[11]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[12]  Ceil Lucas,et al.  Linguistics of American Sign Language: An Introduction , 1995 .

[13]  Ioannis A. Kakadiaris,et al.  Model-based estimation of 3D human motion with occlusion based on active multi-viewpoint selection , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[14]  Ioannis A. Kakadiaris,et al.  3D human body model acquisition from multiple views , 1995, Proceedings of IEEE International Conference on Computer Vision.