Automated Facial Expression Measurement: recent Applications to Basic Research in Human Behavior, Learning, and Education

Automatic facial expression measurement systems have been under development since the early 1990ʼs. The early attempts worked well in highly controlled conditions, but failed for spontaneous expressions, and real application environments. Automatic facial expression recognition has now advanced to the point that we are able to apply it to spontaneous expressions. These systems extract a sufficiently reliable signal that we can employ them in behavioral studies and to begin to develop applications that respond to spontaneous expressions in real time.

[1]  Alex Pentland,et al.  Facial expression recognition using a dynamic model and motion energy , 1995, Proceedings of IEEE International Conference on Computer Vision.

[2]  P. Ekman.  Darwin, Deception, and Facial Expression , 2003, Annals of the New York Academy of Sciences.

[3]  Gwen Littlewort,et al.  Dynamics of Facial Expression Extracted Automatically from Video , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[4]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[5]  Martin A. Giese,et al.  Exploring Human Dynamic Facial Expression Recognition with Animation , 2008 .

[6]  Ashish Kapoor,et al.  Automatic prediction of frustration , 2007, Int. J. Hum. Comput. Stud..

[7]  Wade Junek,et al.  Mind Reading: The Interactive Guide to Emotions , 2007 .

[8]  Christopher J. Fox,et al.  What is adapted in face adaptation? The neural representations of expression in the human visual system , 2007, Brain Research.

[9]  Qiang Ji,et al.  Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  J. Piven,et al.  Abnormal Processing of Social Information from Faces in Autism , 2001, Journal of Cognitive Neuroscience.

[11]  Kenji Mase,et al.  Recognition of Facial Expression from Optical Flow , 1991 .

[12]  Gwen Littlewort,et al.  Automatic Recognition of Facial Actions in Spontaneous Expressions , 2006, J. Multim..

[13]  Ning Wang,et al.  Can Virtual Humans Be More Engaging Than Real Ones? , 2007, HCI.

[14]  Qingshan Liu,et al.  Boosting Coded Dynamic Features for Facial Action Units and Facial Expression Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Raymond Resendes,et al.  Intelligent Vehicle Initiative 2002 Annual Report: Saving Lives Through Advanced Vehicle Safety Technology , 2003 .

[17]  Dimitris N. Metaxas,et al.  Monitoring of facial stress during space flight: Optical computer recognition combining discriminative and generative methods , 2007 .

[18]  Maja Pantic,et al.  Spontaneous vs. posed facial behavior: automatic analysis of brow actions , 2006, ICMI '06.

[19]  W. Rinn,et al.  The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions. , 1984, Psychological bulletin.

[20]  Patrick J. Flynn,et al.  Preliminary Face Recognition Grand Challenge Results , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[21]  Changbo Hu,et al.  AAM derived face representations for robust facial action recognition , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[22]  Marian Stewart Bartlett,et al.  Classifying Facial Actions , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Yair Weiss,et al.  Learning object detection from a small number of examples: the importance of good features , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[24]  Fernando De la Torre,et al.  Temporal Segmentation of Facial Behavior , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[25]  Maja Pantic,et al.  Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[26]  J. Cohn,et al.  Use of Automated Facial Image Analysis for Measurement of Emotion Expression , 2004 .

[27]  M. Holland,et al.  Blinking and Mental Load , 1972, Psychological reports.

[28]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  J. Movellan,et al.  SmileMaze: A Tutoring System in Real-Time Facial Expression Perception and Production in Children with Autism Spectrum Disorder , 2008 .

[30]  Jeffrey R. Spies,et al.  Effects of damping head movement and facial expression in dyadic conversation using real–time facial expression tracking and synthesized avatars , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[31]  Jeffrey R. Spies,et al.  Something in the way we move: Motion dynamics, not perceived sex, influence head movements in conversation. , 2011, Journal of experimental psychology. Human perception and performance.

[32]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[33]  Qiang Ji,et al.  An automated face reader for fatigue detection , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[34]  Jeffrey F. Cohn,et al.  The Timing of Facial Motion in posed and Spontaneous Smiles , 2003, Int. J. Wavelets Multiresolution Inf. Process..

[35]  Zhiwei Zhu,et al.  Dynamic Facial Expression Analysis and Synthesis With MPEG-4 Facial Animation Parameters , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[36]  Liefei Xu April.  THE PROTOTYPE , 1982, The Lancet.

[37]  Gwen Littlewort,et al.  Automatic coding of facial expressions displayed during posed and genuine pain , 2009, Image Vis. Comput..

[38]  Scotty D. Craig,et al.  AutoTutor Detects and Responds to Learners Affective and Cognitive States , 2008 .

[39]  Marian Stewart Bartlett,et al.  Classifying Facial Action , 1995, NIPS.

[40]  R. Gur,et al.  Automated video-based facial expression analysis of neuropsychiatric disorders , 2008, Journal of Neuroscience Methods.

[41]  Maja Pantic,et al.  Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[42]  Hatice Gunes,et al.  How to distinguish posed from spontaneous smiles using geometric features , 2007, ICMI '07.

[43]  Y. Moses,et al.  Real time facial expression recognition in video using support vector machines , 2003, ICMI '03.

[44]  K. Craig,et al.  Genuine, suppressed and faked facial expressions of pain in children , 2006, Pain.

[45]  Matthew S. Goodwin,et al.  Technology for just-in-time in-situ learning of facial affect for persons diagnosed with an autism spectrum disorder , 2008, Assets '08.

[46]  Roddy Cowie.  Building the databases needed to understand rich, spontaneous human behaviour , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[47]  P. Ekman,et al.  Behavioral markers and recognizability of the smile of enjoyment. , 1993, Journal of personality and social psychology.

[48]  Jeffrey F. Cohn,et al.  Automated Facial Image Analysis: Detecting Improvement in Abnormal Facial Movement After Treatment With Botulinum Toxin A , 2007, Annals of plastic surgery.

[49]  James L. Herrick,et al.  Cortical innervation of the facial nucleus in the non-human primate: a new interpretation of the effects of stroke and related subtotal brain trauma on the muscles of facial expression. , 2001, Brain : a journal of neurology.

[50]  Karen L. Schmidt,et al.  Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles , 2003, Biological Psychology.

[51]  Jing Xiao,et al.  Automatic recognition of eye blinking in spontaneously occurring behavior , 2002, Object recognition supported by user interaction for service robots.

[52]  W PicardRosalind,et al.  Toward an Affect-Sensitive AutoTutor , 2007 .

[53]  Rogério Schmidt Feris,et al.  Manifold Based Analysis of Facial Expression , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[54]  Fernando De la Torre,et al.  Facial Expression Analysis , 2011, Visual Analysis of Humans.

[55]  Hideoki Tada.  Eyeblink Rates as a Function of the Interest Value of Video Stimuli , 1986 .

[56]  Gwen Littlewort,et al.  Toward Practical Smile Detection , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[58]  Bo Wu,et al.  Real time facial expression recognition with AdaBoost , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[59]  Algirdas Pakstas,et al.  MPEG-4 Facial Animation: The Standard,Implementation and Applications , 2002 .

[60]  Gwen Littlewort,et al.  A Prototype for Automatic Recognition of Spontaneous Facial Actions , 2002, NIPS.

[61]  Tsuhan Chen,et al.  The painful face - Pain expression recognition using active appearance models , 2009, Image Vis. Comput..

[62]  Larry S. Davis,et al.  Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[63]  Beat Fasel,et al.  Automatic facial expression analysis: a survey , 2003, Pattern Recognit..

[64]  Matthew S. Goodwin,et al.  Developing Innovative Technology for Future Personalized Autism Research and Treatment , 2008 .

[65]  Dimitris N. Metaxas,et al.  Metamorphs: Deformable Shape and Appearance Models , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[66]  Thomas S. Huang,et al.  Final Report To NSF of the Planning Workshop on Facial Expression Understanding , 1992 .

[67]  Peter Robinson,et al.  Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[68]  Gwen Littlewort,et al.  Drowsy Driver Detection Through Facial Movement Analysis , 2007, ICCV-HCI.

[69]  Ian R. Fasel,et al.  A generative framework for real time object detection and classification , 2005, Comput. Vis. Image Underst..

[70]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[71]  T. Sejnowski,et al.  Measuring facial expressions by computer image analysis. , 1999, Psychophysiology.

[72]  A. Anderson,et al.  Examinations of identity invariance in facial expression adaptation , 2008, Cognitive, affective & behavioral neuroscience.

[73]  Thomas S. Huang,et al.  Facial Expression Recognition from Video Sequences : Temporal and Static Modelling , 2002 .

[74]  Dimitris N. Metaxas,et al.  Optical computer recognition of facial expressions associated with stress induced by performance demands. , 2005, Aviation, space, and environmental medicine.

[75]  Gwen Littlewort,et al.  Insights on Spontaneous Facial Expressions from Automatic Expression Measurement , 2010 .