Use of Automated Facial Image Analysis for Measurement of Emotion Expression

Facial expressions are a key index of emotion. They have consistent correlation with self-reported emotion (Keltner, 1995; Rosenberg & Ekman, 1994; Ekman & Rosenberg, in press) and emotion-related central and peripheral physiology (Davidson, Ekman, Saron, Senulis, & Friesen, 1990; Fox & Davidson, 1988; Levenson, Ekman, & Friesen, 1990). They putatively share similar underlying dimensions with self-reported emotion (e.g., positive and negative affect) (Bullock & Russell, 1984; Gross & John, 1997; Watson & Tellegen, 1985). Facial expressions serve interpersonal functions of emotion by conveying communicative intent, signaling affective information in social referencing (Campos, Bertenthal, & Kermoian, 1992), and more generally contributing to the regulation of social interaction (Cohn & Elmore, 1988; Fridlund, 1994; Schmidt & Cohn, 2001). As a measure of trait affect, stability in facial expression emerges early in life (Cohn & Campbell, 1992; Malatesta, Culver, Tesman, & Shephard, 1989). By adulthood, stability is moderately strong, comparable to that for selfreported emotion (Cohn, Schmidt, Gross, & Ekman, 2002), and predictive of favorable outcomes in emotion-related domains including marriage and personal well-being over periods as long as 30 years (Harker & Keltner, 2001). Expressive changes in the face are a rich source of cues about intraand interpersonal functions of emotion (cf. Keltner & Haitd, 1999).

[1]  金出 武雄,et al.  Picture processing system by computer complex and recognition of human faces , 1974 .

[2]  A. Fuchs,et al.  Responses of fibers in medial longitudinal fasciculus (MLF) of alert monkeys during horizontal and vertical conjugate eye movements evoked by vestibular or visual stimuli. , 1976, Journal of neurophysiology.

[3]  R. Kraut,et al.  Social and emotional messages of smiling: An ethological approach. , 1979 .

[4]  Takeo Kanade,et al.  Computer recognition of human faces , 1980 .

[5]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[6]  J. Russell,et al.  Preschool Children's Interpretation of Facial Expressions of Emotion , 1984 .

[7]  D. Watson,et al.  Toward a consensual structure of mood. , 1985, Psychological bulletin.

[8]  J. Cacioppo,et al.  Specific forms of facial EMG response index emotions during an interview: from Darwin to the continuous flow hypothesis affect-laden information processing. , 1988, Journal of personality and social psychology.

[9]  N. Fox,et al.  Patterns of brain electrical activity during facial signs of emotion in 10-month-old infants. , 1988 .

[10]  Jeffrey F. Cohn,et al.  Effect of contingent changes in mothers' affective expression on the organization of behavior in 3-month-old infants , 1988 .

[11]  C. Malatesta,et al.  The development of emotion expression during the first two years of life. , 1989, Monographs of the Society for Research in Child Development.

[12]  R. Matias,et al.  A Comparison of Two Systems that Code Infant Affective Expression. , 1989 .

[13]  P. Ekman,et al.  Voluntary facial action generates emotion-specific autonomic nervous system activity. , 1990, Psychophysiology.

[14]  P. Ekman,et al.  Approach-withdrawal and cerebral asymmetry: emotional expression and brain physiology. I. , 1990, Journal of personality and social psychology.

[15]  U. Hess,et al.  Differentiating emotion elicited and deliberate emotional facial expressions , 1990 .

[16]  Christiana M. Leonard,et al.  When's a Smile a Smile? Or how to Detect a Message by Digitizing the Signal , 1991 .

[17]  George F. Michel,et al.  Infant interest expressions as coordinative motor structures , 1992 .

[18]  Thomas S. Huang,et al.  Final Report To NSF of the Planning Workshop on Facial Expression Understanding , 1992 .

[19]  Susan B. Campbell,et al.  Influence of maternal depression on infant affect regulation. , 1992 .

[20]  Bennett I. Bertenthal,et al.  Early Experience and Emotional Development: The Emergence of Wariness of Heights , 1992 .

[21]  P. Ekman.  Facial expression and emotion. , 1993, The American psychologist.

[22]  P. Ekman,et al.  Behavioral markers and recognizability of the smile of enjoyment. , 1993, Journal of personality and social psychology.

[23]  P. Ekman,et al.  Not all smiles are created equal: the differences between enjoyment and nonenjoyment smiles , 1993 .

[24]  A. J. Fridlund.  Human Facial Expression: An Evolutionary View , 1994 .

[25]  P. Ekman,et al.  Coherence between expressive and experiential systems in emotion , 1994 .

[26]  J. Russell.  Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. , 1994, Psychological bulletin.

[27]  D. Keltner.  Signs of appeasement: evidence for the distinct displays of embarrassment, amusement, and shame , 1995 .

[28]  Larry S. Davis,et al.  Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Dimitris N. Metaxas,et al.  The integration of optical flow and deformable models with applications to human face shape and motion estimation , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[30]  Garrison W. Cottrell,et al.  Representing Face Images for Emotion Classification , 1996, NIPS.

[31]  L. Camras,et al.  Infant “surprise” expressions as coordinative motor structures , 1996 .

[32]  P. Ekman,et al.  The ability to detect deceit generalizes across different types of high-stake lies. , 1997, Journal of personality and social psychology.

[33]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[34]  Mary Target,et al.  Society for Research in Child Development , 1997 .

[35]  O. John,et al.  Revealing feelings: facets of emotional expressivity in self-reports, peer ratings, and behavior. , 1997, Journal of personality and social psychology.

[36]  K. Edwards,et al.  The Face of Time: Temporal Cues in Facial Expressions of Emotion , 1998 .

[37]  Takeo Kanade,et al.  Neural Network-Based Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[39]  U. Dimberg,et al.  Rapid facial reactions to emotional facial expressions. , 1998, Scandinavian journal of psychology.

[40]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[41]  T. Sejnowski,et al.  Measuring facial expressions by computer image analysis. , 1999, Psychophysiology.

[42]  D. Messinger,et al.  What's in a smile? , 1999, Developmental psychology.

[43]  J. Cohn,et al.  Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding. , 1999, Psychophysiology.

[44]  D. Keltner,et al.  Social Functions of Emotions at Four Levels of Analysis , 1999 .

[45]  Zhengyou Zhang,et al.  Feature-Based Facial Expression Recognition: Sensitivity Analysis and Experiments with A Multilayer Perceptron , 1999, Int. J. Pattern Recognit. Artif. Intell..

[46]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[47]  Maja Pantic,et al.  Expert system for automatic analysis of facial expressions , 2000, Image Vis. Comput..

[48]  Beat Fasel,et al.  Recognition of asymmetric facial action unit activities and intensities , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[49]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[50]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[51]  D. Keltner,et al.  PERSONALITY PROCESSES AND INDIVIDUAL DIFFERENCES Expressions of Positive Emotion in Women ' s College Yearbook Pictures and Their Relationship to Personality and Life Outcomes Across Adulthood , 2004 .

[52]  Karen L. Schmidt,et al.  Human facial expressions as adaptations: Evolutionary questions in facial expression research. , 2001, American journal of physical anthropology.

[53]  J. Cohn,et al.  A Psychometric Evaluation of the Facial Action Coding System for Assessing Spontaneous Expression , 2001 .

[54]  Marian Stewart Bartlett,et al.  Computer Recognition of Facial Actions: A study of co-articulation effects , 2001 .

[55]  D. Messinger,et al.  All smiles are positive, but some smiles are more positive than others. , 2001, Developmental psychology.

[56]  Jing Xiao,et al.  Automatic recognition of eye blinking in spontaneously occurring behavior , 2002, Object recognition supported by user interaction for service robots.

[57]  Chi Chung Ko,et al.  Using moment invariants and HMM in facial expression recognition , 2002, Pattern Recognit. Lett..

[58]  D. Messinger.  Positive and Negative: Infant Facial Expressions and Emotions , 2002 .

[59]  Jing Xiao,et al.  Robust full-motion recovery of head by dynamic templates and re-registration techniques , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[60]  Ralph Gross,et al.  Individual differences in facial expression: stability over time, relation to self-reported emotion, and ability to inform person identification , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[61]  Takeo Kanade,et al.  Evaluation of Gabor-wavelet-based facial action unit recognition in image sequences of increasing complexity , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[62]  The Next Generation of Automatic Facial Expression Measurement , 2003 .

[63]  L. Rothkrantz,et al.  Toward an affect-sensitive multimodal human-computer interaction , 2003, Proc. IEEE.

[64]  Karen L. Schmidt,et al.  Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles , 2003, Biological Psychology.

[65]  Jeffrey F. Cohn,et al.  The Timing of Facial Motion in posed and Spontaneous Smiles , 2003, Int. J. Wavelets Multiresolution Inf. Process..

[66]  Hongying Wang,et al.  Three-dimensional eye-head coordination is implemented downstream from the superior colliculus. , 2003, Journal of neurophysiology.

[67]  Thomas S. Huang,et al.  Capturing subtle facial motions in 3D face tracking , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[68]  Yi Zhou,et al.  Bayesian tangent shape model: estimating shape and pose parameters via Bayesian inference , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[69]  Jing Xiao,et al.  Non-rigid shape and motion recovery: degenerate deformations , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[70]  Jing Xiao,et al.  Real-time combined 2D+3D active appearance models , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[71]  Jing Xiao,et al.  Meticulously detailed eye model and its application to analysis of facial image , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[72]  Jing Xiao,et al.  Multimodal coordination of facial action, head rotation, and eye motion during spontaneous smiles , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[73]  Jeffrey F. Cohn,et al.  The Timing of Facial Motion in posed and Spontaneous Smiles , 2003, Int. J. Wavelets Multiresolution Inf. Process..

[74]  Simon Baker,et al.  Active Appearance Models Revisited , 2004, International Journal of Computer Vision.

[75]  Takeo Kanade,et al.  Image Registration Using Wavelet-Based Motion Model , 2000, International Journal of Computer Vision.

[76]  Maja Pantic,et al.  Motion history for facial action detection in video , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[77]  Jing Xiao,et al.  Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[78]  Takahiro Ishikawa,et al.  The template update problem , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[79]  Gwen Littlewort,et al.  Machine learning methods for fully automatic recognition of facial expressions and facial actions , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[80]  R. Scherer,et al.  Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS) , 2005 .