Spontaneous facial expression in a small group can be automatically measured: An initial demonstration

Manual measurement of facial expression is labor intensive and difficult to standardize. Automated measurement seeks to address the need for valid, efficient, and reproducible measurement. Recent systems have shown promise in posed behavior and in structured contexts. Can automated measurement perform in more natural, less constrained settings? In the present study, previously unacquainted young adults sat around a circular table for 30 min of conversation. Video was selected for manual and automatic coding of Facial Action Coding System action units (AUs), examining, in particular, AU 6 (cheek raise) and AU 12 (lip corner pull), which together signal enjoyment. Moderate out-of-plane head motion and occlusion, which are challenging for automatic processing, were both common, as participants turned toward and away from each other or consumed drinks. Concurrent validity for both AUs was high. This is the first study to find that automated measurement of facial action in relatively unconstrained contexts can achieve results comparable with those of manual coding. Video demos of our software may be downloaded from http://brm.psychonomic-journals.org/content/supplemental.

[1]  C. Darwin.  The Expression of the Emotions in Man and Animals , .

[2]  J. Cacioppo,et al.  Electromyographic activity over facial muscle regions can differentiate the valence and intensity of affective reactions. , 1986, Journal of personality and social psychology.

[3]  C. Malatesta,et al.  The development of emotion expression during the first two years of life. , 1989, Monographs of the Society for Research in Child Development.

[4]  P. Ekman,et al.  The Duchenne smile: emotional expression and brain physiology. II. , 1990, Journal of personality and social psychology.

[5]  P. Ekman,et al.  Voluntary facial action generates emotion-specific autonomic nervous system activity. , 1990, Psychophysiology.

[6]  Behavior research methods, instruments, & computers , 1991 .

[7]  R. Larsen,et al.  Promises and problems with the circumplex model of emotion. , 1992 .

[8]  J. Cacioppo,et al.  Primitive emotional contagion. , 1992 .

[9]  P. Ekman.  An argument for basic emotions , 1992 .

[10]  P. Ekman.  Facial expression and emotion. , 1993, The American psychologist.

[11]  A. J. Fridlund.  Human Facial Expression: An Evolutionary View , 1994 .

[12]  J. Russell.  Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. , 1994, Psychological bulletin.

[13]  P. Ekman,et al.  Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique. , 1994, Psychological bulletin.

[14]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  T. Sejnowski,et al.  Measuring facial expressions by computer image analysis. , 1999, Psychophysiology.

[16]  J. Cohn,et al.  Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding. , 1999, Psychophysiology.

[17]  Maja Pantic,et al.  Expert system for automatic analysis of facial expressions , 2000, Image Vis. Comput..

[18]  L P Noldus,et al.  The Observer Video-Pro: New software for the collection, management, and presentation of time-structured data from videotapes and digital media files , 2000, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[19]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Karen L. Schmidt,et al.  Human facial expressions as adaptations: Evolutionary questions in facial expression research. , 2001, American journal of physical anthropology.

[21]  J. Cohn,et al.  A Psychometric Evaluation of the Facial Action Coding System for Assessing Spontaneous Expression , 2001 .

[22]  R. Gibson,et al.  What the Face Reveals , 2002 .

[23]  E. Paluck,et al.  The contingent smile: a meta-analysis of sex differences in smiling. , 2003, Psychological bulletin.

[24]  Takeo Kanade,et al.  Real-time combined 2D+3D active appearance models , 2004, CVPR 2004.

[25]  Amir Averbuch,et al.  Identification of Acoustic Signatures for Vehicles via Reduction of Dimensionality , 2004, Int. J. Wavelets Multiresolution Inf. Process..

[26]  Jeffrey F. Cohn,et al.  The Timing of Facial Motion in posed and Spontaneous Smiles , 2003, Int. J. Wavelets Multiresolution Inf. Process..

[27]  Simon Baker,et al.  Active Appearance Models Revisited , 2004, International Journal of Computer Vision.

[28]  Gwen Littlewort,et al.  Dynamics of Facial Expression Extracted Automatically from Video , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[29]  J. Cohn,et al.  Deciphering the Enigmatic Face , 2005, Psychological science.

[30]  J. Macher,et al.  Facial expressiveness in patients with schizophrenia compared to depressed patients and nonpatient comparison subjects. , 2005, The American journal of psychiatry.

[31]  Dimitris N. Metaxas,et al.  Optical computer recognition of facial expressions associated with stress induced by performance demands. , 2005, Aviation, space, and environmental medicine.

[32]  Yanxi Liu,et al.  The role of structural facial asymmetry in asymmetry of peak facial expressions , 2006, Laterality.

[33]  Thomas R Kirchner,et al.  Effects of alcohol on group formation among male social drinkers. , 2006, Journal of studies on alcohol.

[34]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[35]  Simon Baker,et al.  2D vs. 3D Deformable Face Models: Representational Power, Construction, and Real-Time Fitting , 2007, International Journal of Computer Vision.

[36]  J. Cohn,et al.  Impact of depression on response to comedy: a dynamic facial coding analysis. , 2007, Journal of abnormal psychology.

[37]  Tsuhan Chen,et al.  The painful face - Pain expression recognition using active appearance models , 2009, Image Vis. Comput..

[38]  M. Bartlett,et al.  Machine Analysis of Facial Expressions , 2007 .

[39]  Hatice Gunes,et al.  How to distinguish posed from spontaneous smiles using geometric features , 2007, ICMI '07.

[40]  Chih-Jen Lin,et al.  A Practical Guide to Support Vector Classication , 2008 .

[41]  R. Gur,et al.  Automated video-based facial expression analysis of neuropsychiatric disorders , 2008, Journal of Neuroscience Methods.

[42]  M. Sayette,et al.  Facial reactions to smoking cues relate to ambivalence about smoking. , 2008, Psychology of addictive behaviors : journal of the Society of Psychologists in Addictive Behaviors.

[43]  J. Cohn,et al.  All Smiles are Not Created Equal: Morphology and Timing of Smiles Perceived as Amused, Polite, and Embarrassed/Nervous , 2009, Journal of nonverbal behavior.

[44]  Gwen Littlewort,et al.  Automatic coding of facial expressions displayed during posed and genuine pain , 2009, Image Vis. Comput..

[45]  Gwen Littlewort,et al.  Toward Practical Smile Detection , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Jeffrey F. Cohn,et al.  Facial image synthesis. , 2009 .

[47]  Simon Lucey,et al.  Automated Facial Expression Recognition System , 2009, 43rd Annual 2009 International Carnahan Conference on Security Technology.

[48]  C. Darwin,et al.  The Expression of the Emotions in Man and Animals , 1956 .

[49]  J. Cohn,et al.  Automated Measurement of Facial Expression in Infant-Mother Interaction: A Pilot Study. , 2009, Infancy : the official journal of the International Society on Infant Studies.

[50]  Fernando De la Torre,et al.  Detecting depression from facial actions and vocal prosody , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[51]  Tsuhan Chen,et al.  The painful face - Pain expression recognition using active appearance models , 2009, Image Vis. Comput..