Toward computational understanding of sign language

In this paper, we describe some of the current issues in computational sign language processing. Despite the seeming similarities between computational spoken language and sign language processing, signed languages have intrinsic properties that pose some very difficult problems. These include a high level of simultaneous actions, the intersection between signs and gestures, and the complexity of modeling grammatical processes. Additional problems are posed by the difficulties that computers face in extracting reliable information on the hands and the face from video images. So far, no single research group or company has managed to tackle all the hard problems and produced a real working system for analysis and recognition. We present a summary of our research into sign language recognition and how it interacts with sign language linguistics. We propose solutions to some of the aforementioned problems, and also discuss what problems are still unsolved. In addition, we summarize the current state of the art in our sign language recognition and facial expression analysis frameworks.

[1]  Dimitris N. Metaxas,et al.  Parallel hidden Markov models for American sign language recognition , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[2]  Dimitris N. Metaxas,et al.  Toward Scalability in ASL Recognition: Breaking Down Signs into Phonemes , 1999, Gesture Workshop.

[3]  Dimitris N. Metaxas,et al.  Handshapes and movements: Multiple-channel ASL recognition , 2004 .

[4]  Ming Ouhyoung,et al.  A real-time continuous gesture recognition system for sign language , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[5]  Thomas Hopkins,et al.  What is American Sign Language , 2009 .

[6]  Wen Gao,et al.  A Real-Time Large Vocabulary Recognition System for Chinese Sign Language , 2001, Gesture Workshop.

[7]  S. F. Taub,et al.  Patterns of Conceptual Encoding in ASL Motion Descriptions , 2001 .

[8]  Surendra Ranganath,et al.  Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[10]  Dimitris N. Metaxas,et al.  A Framework for Recognizing the Simultaneous Aspects of American Sign Language , 2001, Comput. Vis. Image Underst..

[11]  Luiz Velho,et al.  Adaptive Deformable Models for Graphics and Vision † , 2005, Comput. Graph. Forum.

[12]  R. Battison,et al.  Lexical Borrowing in American Sign Language , 1978 .

[13]  Kuldip K. Paliwal,et al.  Automatic Speech and Speaker Recognition , 1996 .

[14]  Dimitris N. Metaxas,et al.  Learning-based dynamic coupling of discrete and continuous trackers , 2006, Comput. Vis. Image Underst..

[15]  Diane Brentari,et al.  A Prosodic Model of Sign Language Phonology , 1999 .

[16]  Kuldip K. Paliwal,et al.  Automatic Speech and Speaker Recognition: Advanced Topics , 1999 .

[17]  Dimitris N. Metaxas,et al.  Adjusting Shape Parameters Using Model-Based Optical Flow Residuals , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[19]  Karl-Friedrich Kraiss,et al.  Video-based sign recognition using self-organizing subunits , 2002, Object recognition supported by user interaction for service robots.

[20]  Scott K. Liddell,et al.  American Sign Language: The Phonological Base , 2013 .

[21]  Karl-Friedrich Kraiss,et al.  Towards an Automatic Sign Language Recognition System Using Subunits , 2001, Gesture Workshop.

[22]  Siome Goldenstein,et al.  Statistical Cue Integration in DAG Deformable Models , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Stan Sclaroff,et al.  Estimating 3D hand pose from a cluttered image , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[24]  William H. Edmondson,et al.  International review of sign linguistics , 1989 .

[25]  Ceil Lucas,et al.  Linguistics of American Sign Language: An Introduction , 1995 .

[26]  W. Sandler.  Phonological Representation of the Sign: Linearity and Nonlinearity in American Sign Language , 1989 .

[27]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[29]  Siome Goldenstein,et al.  3D facial tracking from corrupted movie sequences , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[30]  Siome Goldenstein,et al.  The Best of Both Worlds: Combining 3D Deformable Models with Active Shape Models , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[31]  W. Stokoe,et al.  Sign language structure: an outline of the visual communication systems of the American deaf. 1960. , 1961, Journal of deaf studies and deaf education.

[32]  Jorge Stolfi,et al.  Outlier rejection in high-dimensional deformable models , 2007, Image Vis. Comput..