Extraction of texture and geometrical features from informative facial regions for sign language recognition

In general, the most common form of gestures is made up of movements of the hand and/or arm associated with facial expressions. In this, the hand is used to make different message signs, while facial movements are used to reflect the mood and emotion of the person. In this paper, some sign language gestures are recognized only with the help of associated facial expressions. Existing facial expression based sign language recognition (SLR) methods only used facial geometric features to recognize sign language gestures. However, the performance of geometric feature-based SLR methods depends on the accuracy of tracking algorithms and the number of facial landmark points. Additionally, facial textures are more informative as compared to the geometric features of a face. Inspiring from these facts, we propose to recognize sign language gestures with the help of spatio-temporal characteristics of facial texture patterns. For this, a new face model is proposed by extracting texture features only from the informative regions of a face. The proposed face model can also be employed to extract the geometrical features of a face. The features extracted from the informative regions of a face are significantly discriminative, and so the proposed face model can track/encode the facial dynamics of the associated facial expressions of a sign. Finally, a 3-state hidden conditional random field is employed to model the texture variations of facial gestures. Experimental results on RWTH-BOSTON data-set show that proposed method can achieve upto 80.06% recognition rate.

[1]  Dimitris N. Metaxas,et al.  Parallel hidden Markov models for American sign language recognition , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[2]  Wen Gao,et al.  A vision-based sign language recognition system using tied-mixture density HMM , 2004, ICMI '04.

[3]  Anastasios Delopoulos,et al.  The MUG facial expression database , 2010, 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10.

[4]  Karl-Friedrich Kraiss,et al.  Towards a Video Corpus for Signer-Independent Continuous Sign Language Recognition , 2007 .

[5]  Alex Pentland,et al.  Coupled hidden Markov models for complex action recognition , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[6]  Maja Pantic,et al.  Coupled Gaussian processes for pose-invariant facial expression recognition , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Moritz Knorr,et al.  The significance of facial features for automatic sign language recognition , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[8]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  David J. Kriegman,et al.  Localizing parts of faces using a consensus of exemplars , 2011, CVPR.

[10]  Wen Gao,et al.  Large-Vocabulary Continuous Sign Language Recognition Based on Transition-Movement Models , 2007, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[11]  Michael J. Lyons,et al.  Automatic Classification of Single Facial Images , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Vinod Chandran,et al.  Evaluation of Texture and Geometry for Dimensional Facial Expression Recognition , 2011, 2011 International Conference on Digital Image Computing: Techniques and Applications.

[13]  Di Huang,et al.  Local Binary Patterns and Its Application to Facial Image Analysis: A Survey , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[14]  Elisabeth Engberg-Pedersen,et al.  Space in Danish sign language : the semantics and morphosyntax of the use of space in a visual language , 1993 .

[15]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.

[16]  Surendra Ranganath,et al.  Facial expressions in American sign language: Tracking and recognition , 2012, Pattern Recognit..

[17]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[18]  Hermann Ney,et al.  Combination of Tangent Distance and an Image Distortion Model for Appearance-Based Sign Language Recognition , 2005, DAGM-Symposium.

[19]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[20]  Aurobinda Routray,et al.  Automatic facial expression recognition using features of salient facial patches , 2015, IEEE Transactions on Affective Computing.

[21]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[23]  Seong-Whan Lee,et al.  Combination of manual and non-manual features for sign language recognition based on conditional random field and active appearance model , 2011, 2011 International Conference on Machine Learning and Cybernetics.

[24]  Trevor Darrell,et al.  Hidden Conditional Random Fields for Gesture Recognition , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[25]  Vladimir Pavlovic,et al.  Hidden Conditional Ordinal Random Fields for Sequence Classification , 2010, ECML/PKDD.

[26]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[27]  Vladimir Pavlovic,et al.  Variable-state latent conditional random fields for facial expression recognition and action unit detection , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[28]  Maja Pantic,et al.  Optimization Problems for Fast AAM Fitting in-the-Wild , 2013, 2013 IEEE International Conference on Computer Vision.

[29]  P. Ekman,et al.  Facial action coding system , 2019 .

[30]  Raphael C.-W. Phan,et al.  Facial Expression Recognition in the Encrypted Domain Based on Local Fisher Discriminant Analysis , 2013, IEEE Transactions on Affective Computing.

[31]  Surendra Ranganath,et al.  Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  J. Turner.  Human Emotions: A Sociological Theory , 2007 .

[33]  Surendra Ranganath,et al.  Tracking facial features under occlusions and recognizing facial expressions in sign language , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[34]  Ismail Ari,et al.  Facial feature tracking and expression recognition for sign language , 2009, 2009 IEEE 17th Signal Processing and Communications Applications Conference.