A Multi-Method Approach for Discriminating Between Similar Facial Expressions, Including Expression Intensity Estimation

Facial expression provides sensitive cues about emotion and plays a major role in interpersonal and humancomputer interaction. Most facial expression recognition systems have focused on only six basic emotions and their concomitant prototypic expressions posed by a small set of subjects. In reality, humans are capable of producing thousands of expressions that vary in complexity, intensity, and meaning. To represent the full range of facial expression, we developed a computer vision system that automatically recognizes individual action units (AUs) or AU combinations using Hidden Markov Models and estimates expression intensity. Three modules are used to extract facial expression information: (1) facial feature point tracking, (2) dense flow tracking with principal component analysis (PCA), and (3) high gradient component detection (i.e. furrow detection). The average recognition rate of upper and lower face expressions is 85% and 88%, respectively, using feature point tracking, 93% (upper face) using dense flow tracking with PCA, and 85% and 81%, upper and lower face respectively, using high gradient component detection.

[1]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[2]  J. N. Bassili.  Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face. , 1979, Journal of personality and social psychology.

[3]  Robert M. Gray,et al.  An Algorithm for Vector Quantizer Design , 1980, IEEE Trans. Commun..

[4]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[5]  John Nolt.  EXPRESSION AND EMOTION , 1981 .

[6]  W. Rinn,et al.  The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions. , 1984, Psychological bulletin.

[7]  D. McNeill.  So you think gestures are nonverbal , 1985 .

[8]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[9]  Jeffrey F. Cohn,et al.  Effect of contingent changes in mothers' affective expression on the organization of behavior in 3-month-old infants , 1988 .

[10]  Alex Pentland,et al.  Automatic lipreading by optical-flow analysis , 1989 .

[11]  Demetri Terzopoulos,et al.  Analysis of facial images using physical and anatomical models , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[12]  Lawrence Sirovich,et al.  Application of the Karhunen-Loeve Procedure for the Characterization of Human Faces , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[14]  Kenji Mase,et al.  Recognition of Facial Expression from Optical Flow , 1991 .

[15]  P. Ekman.  Facial expression and emotion. , 1993, The American psychologist.

[16]  A. J. Fridlund.  Human Facial Expression: An Evolutionary View , 1994 .

[17]  Larry S. Davis,et al.  Computing spatio-temporal representations of human faces , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Jie Yang.  Hidden markov model for human performance modeling , 1994 .

[19]  M. Rosenblum,et al.  Human emotion recognition from motion using a radial basis function network architecture , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[20]  Irfan Essa,et al.  Analysis, interpretation and synthesis of facial expressions , 1995 .

[21]  Marian Stewart Bartlett,et al.  Classifying Facial Action , 1995, NIPS.

[22]  Michael J. Black,et al.  Recognizing facial expressions under rigid and non-rigid facial motions , 1995 .

[23]  David J. Fleet,et al.  Learning parameterized models of image motion , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[24]  Takeo Kanade,et al.  Optical flow estimation using wavelet motion model , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).