Local binary pattern based features for sign language recognition

In this paper we focus on appearance features describing the manual component of Sign Language particularly the Local Binary Patterns. We compare the performance of these features with geometric moments describing the trajectory and shape of hands. Since the non-manual component is also very important for sign recognition we localize facial landmarks via Active Shape Model combined with Landmark detector that increases the robustness of model fitting. We test the recognition performance of individual features and their combinations on a database consisting of 11 signers and 23 signs with several repetitions. Local Binary Patterns outperform the geometric moments. When the features are combined we achieve a recognition rate up to 99.75% for signer dependent tests and 57.54% for signer independent tests.

[1]  Surendra Ranganath,et al.  Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Moritz Knorr,et al.  The significance of facial features for automatic sign language recognition , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[3]  Timothy F. Cootes,et al.  Boosted Regression Active Shape Models , 2007, BMVC.

[4]  Marek Hrúz,et al.  Semi-automatic Annotation of Sign Language Corpora , 2008 .

[5]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[6]  Wen Gao,et al.  A Real-Time Large Vocabulary Continuous Recognition System for Chinese Sign Language , 2001, IEEE Pacific Rim Conference on Multimedia.

[7]  Daniel Kelly,et al.  A person independent system for recognition of hand postures used in sign language , 2010, Pattern Recognit. Lett..

[8]  Fred Nicolls,et al.  Locating Facial Features with an Extended Active Shape Model , 2008, ECCV.

[9]  Kanis Jakub,et al.  Towards Czech on-line sign language dictionary - technological overview and data collection , 2010, LREC 2010.

[10]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[11]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Ludek Müller,et al.  Czech Text-to-Sign Speech Synthesizer , 2007, MLMI.

[13]  Jürgen Beyerer,et al.  Fast Invariant Contour-Based Classification of Hand Symbols for HCI , 2009, CAIP.

[14]  Karl-Friedrich Kraiss,et al.  Sign Language Recognition , 2006 .

[15]  Jan Zelinka,et al.  Feature space transforms for Czech sign-language recognition , 2008, INTERSPEECH.

[16]  Lale Akarun,et al.  Sign Language Tutoring tool , 2005, 2005 13th European Signal Processing Conference.

[17]  Wu-Chih Hu,et al.  Vision-Based Hand Gesture Recognition Using PCA+Gabor Filters and SVM , 2009, 2009 Fifth International Conference on Intelligent Information Hiding and Multimedia Signal Processing.

[18]  Wen Gao,et al.  A Real-Time Large Vocabulary Recognition System for Chinese Sign Language , 2001, Gesture Workshop.

[19]  Marcel J. T. Reinders,et al.  Influence of Handshape Information on Automatic Sign Language Recognition , 2009, Gesture Workshop.