Viewpoint invariant sign language recognition

Sign language is the primary modality of communication among deaf and mute society all over the world. This paper proposes a viewpoint independent method for sign recognition. Considering that two sequences of the same sign can be roughly considered as the input of a stereo vision system after time-warping, and the fundamental matrix associated with two views should be unique, we can convert the temporal-spatial recognition task as a verification task within a stereo vision framework. After time-warping of the input sequences, the proposed framework can reach both temporal and viewpoint invariance. We demonstrate the efficiency of the proposed framework by recognizing a vocabulary of 100 words of Chinese sign language. The recognition rate is up to 97% at rank 3. Furthermore, the proposed framework can be easily extended to other recognition tasks, such as gait recognition and lip-reading recognition.

[1]  Yaser Sheikh,et al.  On the use of anthropometry in the invariant analysis of human actions , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[2]  George Kollios,et al.  BoostMap: A method for efficient approximate similarity rankings , 2004, CVPR 2004.

[3]  Yoshiaki Shirai,et al.  Extraction of Hand Features for Recognition of Sign Language Words , 2002 .

[4]  M. Shah,et al.  On the use of anthropometry in the invariant analysis of human actions , 2004, ICPR 2004.

[5]  Hermann Hienz,et al.  Video-based continuous sign language recognition using statistical methods , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[6]  Jochen Triesch,et al.  Robust classification of hand postures against complex backgrounds , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[7]  John K. Tsotsos,et al.  Hand Gesture Recognition within a Linguistics-Based Framework , 2004, ECCV.

[8]  Dimitris N. Metaxas,et al.  ASL recognition based on a coupling between HMMs and 3D motion analysis , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[9]  Narendra Ahuja,et al.  Extraction of 2D Motion Trajectories and Its Application to Hand Gesture Recognition , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Dimitris N. Metaxas,et al.  A Framework for Recognizing the Simultaneous Aspects of American Sign Language , 2001, Comput. Vis. Image Underst..

[11]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[12]  Ying Wu,et al.  Vision-Based Gesture Recognition: A Review , 1999, Gesture Workshop.

[13]  Kirsti Grobel,et al.  Video-Based Sign Language Recognition Using Hidden Markov Models , 1997, Gesture Workshop.

[14]  Quang-Tuan Luong,et al.  Self-Calibration of a Moving Camera from Point Correspondences and Fundamental Matrices , 1997, International Journal of Computer Vision.

[15]  Lalit Gupta,et al.  Gesture-based interaction and communication: automated classification of hand gesture contours , 2001, IEEE Trans. Syst. Man Cybern. Syst..

[16]  David Windridge,et al.  A Linguistic Feature Vector for the Visual Interpretation of Sign Language , 2004, ECCV.

[17]  Hermann Hienz,et al.  Relevant features for video-based continuous sign language recognition , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[18]  Wen Gao,et al.  A vision-based sign language recognition system using tied-mixture density HMM , 2004, ICMI '04.

[19]  Mansoor Sarhadi,et al.  A non-linear model of shape and motion for tracking finger spelt American sign language , 2002, Image Vis. Comput..

[20]  Karl-Friedrich Kraiss,et al.  Towards an Automatic Sign Language Recognition System Using Subunits , 2001, Gesture Workshop.

[21]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[22]  Ying Wu,et al.  View-independent recognition of hand postures , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[23]  Hyeonjoon Moon,et al.  The FERET Evaluation Methodology for Face-Recognition Algorithms , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Karl-Friedrich Kraiss,et al.  Video-based sign recognition using self-organizing subunits , 2002, Object recognition supported by user interaction for service robots.

[25]  Andrew W. Fitzgibbon,et al.  Real-time gesture recognition using deterministic boosting , 2002, BMVC.

[26]  Vladimir Pavlovic,et al.  Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Thad Starner,et al.  Visual Recognition of American Sign Language Using Hidden Markov Models. , 1995 .

[28]  Surendra Ranganath,et al.  Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Tanveer F. Syeda-Mahmood,et al.  View-invariant alignment and matching of video sequences , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[30]  Omar M. Al-Jarrah,et al.  Recognition of gestures in Arabic sign language using neuro-fuzzy systems , 2001, Artif. Intell..

[31]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Yuntao Cui,et al.  Appearance-Based Hand Sign Recognition from Intensity Image Sequences , 2000, Comput. Vis. Image Underst..