Enhancing gloss-based corpora with facial features using active appearance models

In text-to-avatar translation systems, facial expressions and mouth patterns are a vital part of a natural sign language avatar animation. However, gloss based corpora often lack detail with respect to such non-manual features. To create a translation system which can produce facial expressions and mouthings, a more fine-grained annotation is necessary. In this work, we apply a clustering algorithm to automatically distinguish between different facial patterns using an active appearance model. The resulting translation system is then able to produce such expressions based on the written language text. In our experiments, the system produced suitable expressions with an accuracy of 78.37%.

[1]  Ralph Gross,et al.  Generic vs. person specific active appearance models , 2005, Image Vis. Comput..

[2]  Vassilis Pitsikalis,et al.  Experiments on Global and Local Active Appearance Models for Analysis of Sign Language Facial Expressions , 2011 .

[3]  Georg Heigold,et al.  The RWTH aachen university open source speech recognition system , 2009, INTERSPEECH.

[4]  Hermann Ney,et al.  A Systematic Comparison of Various Statistical Alignment Models , 2003, CL.

[5]  Hermann Ney,et al.  Analysis, preparation, and optimization of statistical sign language machine translation , 2012, Machine Translation.

[6]  Timothy F. Cootes,et al.  Interpreting face images using active appearance models , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[7]  Karl-Friedrich Kraiss,et al.  Recent developments in visual sign language recognition , 2008, Universal Access in the Information Society.

[8]  Andrew Rosenberg,et al.  Evaluating importance of facial expression in american sign language and pidgin signed english animations , 2011, ASSETS.

[9]  Ismail Ari,et al.  Facial feature tracking and expression recognition for sign language , 2009, 2009 IEEE 17th Signal Processing and Communications Applications Conference.

[10]  Alexis Héloir,et al.  Assessing the deaf user perspective on sign language avatars , 2011, ASSETS.

[11]  Simon Lucey,et al.  Real-time avatar animation from a single image , 2011, Face and Gesture 2011.

[12]  Simon Baker,et al.  Active Appearance Models Revisited , 2004, International Journal of Computer Vision.

[13]  Hermann Ney,et al.  RWTH-PHOENIX-Weather: A Large Vocabulary Sign Language Recognition and Translation Corpus , 2012, LREC.

[14]  Wei Du,et al.  Video analysis for continuous sign language recognition , 2010 .

[15]  Takeo Kanade,et al.  Real-time combined 2D+3D active appearance models , 2004, CVPR 2004.

[16]  Franck Davoine,et al.  Facial expression recognition and synthesis based on an appearance model , 2004, Signal Process. Image Commun..

[17]  Naveed Iqbal Rao,et al.  Color Image Segmentation Using Multilevel Clustering Approach , 2008, 2008 Digital Image Computing: Techniques and Applications.

[18]  Great Britain. Hm Factory Inspectorate.  An introductory guide , 1987 .

[19]  Trevor Johnston.  The lexical database of Auslan (Australian Sign Language) , 2002 .