Continuous Profile Models in ASL Syntactic Facial Expression Synthesis

To create accessible content for deaf users, we investigate automatically synthesizing animations of American Sign Language (ASL), including grammatically important facial expressions and head movements. Based on recordings of humans performing various types of syntactic face and head movements (which include idiosyncratic variation), we evaluate the efficacy of Continuous Profile Models (CPMs) at identifying an essential “latent trace” of the performance, for use in producing ASL animations. A metric-based evaluation and a study with deaf users indicated that this approach was more effective than a prior method for producing animations.

[1]  Carol Neidle,et al.  Centroid-Based Exemplar Selection of ASL Non-Manual Expressions using Multidimensional Dynamic Time Warping and MPEG4 Features , 2016 .

[2]  Michael Filhol,et al.  A rule triggering system for automatic text-to-sign translation , 2016, Universal Access in the Information Society.

[3]  Matt Huenerfauth,et al.  Measuring the Perception of Facial Expressions in American Sign Language Animations with Eye Tracking , 2014, HCI.

[4]  Hermann Ney,et al.  Enhancing gloss-based corpora with facial features using active appearance models , 2013 .

[5]  Matt Huenerfauth,et al.  Effect of Displaying Human Videos During an Evaluation Study of American Sign Language Animation , 2013, TACC.

[6]  John Glauert,et al.  Exploiting the full potential of JASigning to build an avatar signing train announcements , 2013 .

[7]  Amit K. Roy-Chowdhury,et al.  Face Tracking , 2009, Encyclopedia of Biometrics.

[8]  Hernisa Kacorri,et al.  TR-2015001: A Survey and Critique of Facial Expression Synthesis in Sign Language Animation , 2015 .

[9]  Nicolas Courty,et al.  The SignCom system for data-driven animation of interactive virtual signers , 2011, ACM Trans. Interact. Intell. Syst..

[10]  A. Poritz,et al.  Hidden Markov models: a guided tour , 1988, ICASSP-88., International Conference on Acoustics, Speech, and Signal Processing.

[11]  G. Dunteman.  Principal Components Analysis , 1989 .

[12]  J. A. López del Val,et al.  Principal Components Analysis , 2018, Applied Univariate, Bivariate, and Multivariate Statistics Using Python.

[13]  Matt Huenerfauth,et al.  Evaluating Facial Expressions in American Sign Language Animations for Accessible Online Information , 2013, HCI.

[14]  Matt Huenerfauth.  Augmenting EMBR Virtual Human Animation System with MPEG-4 Controls for Producing ASL Facial Expressions , 2015 .

[15]  Dimitris N. Metaxas,et al.  Computer-based tracking, analysis, and visualization of linguistically significant nonmanual events in American Sign Language (ASL) , 2014 .

[16]  Alexis Héloir,et al.  EMBR: A realtime animation engine for interactive embodied agents , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[17]  Matt Huenerfauth,et al.  Demographic and Experiential Factors Influencing Acceptance of Sign Language Animation by Deaf Users , 2015, ASSETS.

[18]  Charlotte Baker-Shenk,et al.  A Microanalysis of the Nonmanual Components of Questions in American Sign Language , 1983 .

[19]  Radford M. Neal,et al.  Multiple Alignment of Continuous Time Series , 2004, NIPS.

[20]  Hermann Ney,et al.  Analysis, preparation, and optimization of statistical sign language machine translation , 2012, Machine Translation.

[21]  Matt Huenerfauth,et al.  Evaluating a Dynamic Time Warping Based Scoring Algorithm for Facial Expressions in ASL Animations , 2015, SLPAT@Interspeech.

[22]  J. Listgarten.  Analysis of sibling time series data: Alignment and difference detection , 2007 .

[23]  Matt Huenerfauth,et al.  Best practices for conducting evaluations of sign language animation , 2015 .