Robust sign language recognition by combining manual and non-manual features based on conditional random field and support vector machine

The sign language is composed of two categories of signals: manual signals such as signs and fingerspellings and non-manual ones such as body gestures and facial expressions. This paper proposes a new method for recognizing manual signals and facial expressions as non-manual signals. The proposed method involves the following three steps: First, a hierarchical conditional random field is used to detect candidate segments of manual signals. Second, the BoostMap embedding method is used to verify hand shapes of segmented signs and to recognize fingerspellings. Finally, the support vector machine is used to recognize facial expressions as non-manual signals. This final step is taken when there is some ambiguity in the previous two steps. The experimental results indicate that the proposed method can accurately recognize the sign language at an 84% rate based on utterance data.

[1]  Lale Akarun,et al.  A belief-based sequential fusion approach for fusing manual signs and non-manual signals , 2009, Pattern Recognit..

[2]  Surendra Ranganath,et al.  Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Surendra Ranganath,et al.  Representations for facial expressions , 2002, 7th International Conference on Control, Automation, Robotics and Vision, 2002. ICARCV 2002..

[4]  Wei-Yang Lin,et al.  Kernel-based representation for 2D/3D motion trajectory retrieval and classification , 2013, Pattern Recognit..

[5]  Aditi Lahiri,et al.  Height Differences in English Dialects: Consequences for Processing and Representation , 2010, Language and speech.

[6]  Thad Starner,et al.  American sign language recognition with the kinect , 2011, ICMI '11.

[7]  Seong-Whan Lee,et al.  Simultaneous spotting of signs and fingerspellings based on hierarchical conditional random fields and boostmap embeddings , 2010, Pattern Recognit..

[8]  R. Wilbur.  Effects of Varying Rate of Signing on ASL Manual Signs and Nonmanual Markers , 2009, Language and speech.

[9]  Wen Gao,et al.  A Parallel Multistream Model for Integration of Sign Language Recognition and Lip Motion , 2000, ICMI.

[10]  Ayoub Al-Hamadi,et al.  A Robust Method for Hand Gesture Segmentation and Recognition Using Forward Spotting Scheme in Conditional Random Fields , 2010, 2010 20th International Conference on Pattern Recognition.

[11]  Stan Sclaroff,et al.  Sign Language Spotting with a Threshold Model Based on Conditional Random Fields , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Nicolas Pugeault,et al.  Sign language recognition using sub-units , 2012, J. Mach. Learn. Res..

[13]  Seong-Whan Lee,et al.  Combination of manual and non-manual features for sign language recognition based on conditional random field and active appearance model , 2011, 2011 International Conference on Machine Learning and Cybernetics.

[14]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Stephan Liwicki,et al.  Automatic recognition of fingerspelled words in British Sign Language , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.