Recognition of facial expressions and measurement of levels of interest from video

This paper presents a spatio-temporal approach in recognizing six universal facial expressions from visual data and using them to compute levels of interest. The classification approach relies on a two-step strategy on the top of projected facial motion vectors obtained from video sequences of facial expressions. First a linear classification bank was applied on projected optical flow vectors and decisions made by the linear classifiers were coalesced to produce a characteristic signature for each universal facial expression. The signatures thus computed from the training data set were used to train discrete hidden Markov models (HMMs) to learn the underlying model for each facial expression. The performances of the proposed facial expressions recognition were computed using five fold cross-validation on Cohn-Kanade facial expressions database consisting of 488 video sequences that includes 97 subjects. The proposed approach achieved an average recognition rate of 90.9% on Cohn-Kanade facial expressions database. Recognized facial expressions were mapped to levels of interest using the affect space and the intensity of motion around apex frame. Computed level of interest was subjectively analyzed and was found to be consistent with "ground truth" information in most of the cases. To further illustrate the efficacy of the proposed approach, and also to better understand the effects of a number of factors that are detrimental to the facial expression recognition, a number of experiments were conducted. The first empirical analysis was conducted on a database consisting of 108 facial expressions collected from TV broadcasts and labeled by human coders for subsequent analysis. The second experiment (emotion elicitation) was conducted on facial expressions obtained from 21 subjects by showing the subjects six different movies clips chosen in a manner to arouse spontaneous emotional reactions that would produce natural facial expressions.

[1]  A. Anderson,et al.  The Effects of Visibility on Dialogue and Performance in a Cooperative Problem Solving Task , 1994 .

[2]  CHARLES A. NELSON,et al.  Asymmetry in Facial Expression The conclusion of Sackeim , 2005 .

[3]  Alex Pentland,et al.  LAFTER: Lips and Face Real Time Tracker with Facial Expression Recognition , 1997, CVPR 1997.

[4]  Paul Ekman,et al.  Emotions inside out. 130 Years after Darwin's "The Expression of the Emotions in Man and Animal". , 2003, Annals of the New York Academy of Sciences.

[5]  Michael J. Black,et al.  Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion , 1995, Proceedings of IEEE International Conference on Computer Vision.

[6]  A. Pentland,et al.  Attention-driven Expression and Gesture Analysis in an Interactive Environment , 1995 .

[7]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Mohammed Yeasin,et al.  Detecting and tracking human face and eye using an space-varying sensor and an active vision head , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[9]  Andrew Blake,et al.  Determining facial expressions in real time , 1995, Proceedings of IEEE International Conference on Computer Vision.

[10]  A. Treves,et al.  A neural network facial expression recognition system using unsupervised local processing , 2001, ISPA 2001. Proceedings of the 2nd International Symposium on Image and Signal Processing and Analysis. In conjunction with 23rd International Conference on Information Technology Interfaces (IEEE Cat..

[11]  E. Fox,et al.  The face of fear: Effects of eye gaze and emotion on visual attention , 2003, Visual cognition.

[12]  John Nolt.  EXPRESSION AND EMOTION , 1981 .

[13]  D. Rutter,et al.  The Role of Visual Communication in Social Exchange , 1976 .

[14]  Yasunari Yoshitomi,et al.  Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face , 2000, Proceedings 9th IEEE International Workshop on Robot and Human Interactive Communication. IEEE RO-MAN 2000 (Cat. No.00TH8499).

[15]  P. Ekman,et al.  Emotion, physiology, and expression in old age. , 1991, Psychology and aging.

[16]  Michael J. Black,et al.  Recognizing Facial Expressions in Image Sequences Using Local Parameterized Models of Image Motion , 1997, International Journal of Computer Vision.

[17]  P. Ekman.  Pictures of Facial Affect , 1976 .

[18]  Tomaso A. Poggio,et al.  Learning-based approach to real time tracking and analysis of faces , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[19]  P. Ekman,et al.  BODY POSITION, FACIAL EXPRESSION, AND VERBAL BEHAVIOR DURING INTERVIEWS. , 1964, Journal of abnormal psychology.

[20]  P. Ekman.  Are there basic emotions? , 1992, Psychological review.

[21]  Rosalind W. Picard.  Toward Agents that Recognize Emotion , 1998 .

[22]  Takeo Kanade,et al.  Neural Network-Based Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Garrison W. Cottrell,et al.  Representing Face Images for Emotion Classification , 1996, NIPS.

[24]  Human emotion. , 1978, Nebraska Symposium on Motivation. Nebraska Symposium on Motivation.

[25]  Mohammed Yeasin,et al.  Tracking the human arm using constraint fusion and multiple-cue localization , 2003, Machine Vision and Applications.

[26]  F. D. Horowitz,et al.  Asymmetry in facial expression. , 1980, Science.

[27]  Mark Steedman,et al.  Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents , 1994, SIGGRAPH.

[28]  P. Ekman,et al.  DIFFERENCES Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion , 2004 .

[29]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[30]  Ioannis Pavlidis,et al.  The face of fear , 2001 .

[31]  R. R. Avent,et al.  Machine vision recognition of facial affect using backpropagation neural networks , 1994, Proceedings of 16th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[32]  Alex Pentland,et al.  Facial expression recognition using a dynamic model and motion energy , 1995, Proceedings of IEEE International Conference on Computer Vision.

[33]  P. Ekman,et al.  Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique. , 1994, Psychological bulletin.

[34]  Christine L. Lisetti,et al.  Facial Expression Recognition Using a Neural Network , 1998, FLAIRS.

[35]  Rosalind W. Picard.  Toward computers that recognize and respond to user emotion , 2000, IBM Syst. J..

[36]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[37]  Paul Ekman.  Expression: panel discussion. , 2003, Annals of the New York Academy of Sciences.

[38]  M. Rosenblum,et al.  Human emotion recognition from motion using a radial basis function network architecture , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[39]  C. Breazeal.  Sociable Machines: Expressive Social Ex-change Between Humans and Robots , 2000 .

[40]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[41]  Yasuhiro Nakamura,et al.  Mathematical representation and image generation of human faces by metamorphosis , 1997 .

[42]  Rosalind W. Picard,et al.  Finding similar patterns in large image databases , 1993, 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[43]  Christine L. Lisetti,et al.  Automatic facial expression interpretation: Where human-computer interaction, artificial intelligence and cognitive science intersect , 2000 .

[44]  Alex Pentland,et al.  LAFTER: a real-time face and lips tracker with facial expression recognition , 2000, Pattern Recognit..

[45]  Thomas S. Huang,et al.  Emotion Recognition from Facial Expressions using Multilevel HMM , 2000 .

[46]  Cynthia Breazeal,et al.  Robot in Society: Friend or Appliance? , 1999 .

[47]  Alex Pentland,et al.  Correlation and Interpolation Networks for Real-time Expression Analysis/Synthesis , 1994, NIPS.

[48]  D. Keltner,et al.  Culture and Facial Expression: Open-ended Methods Find More Expressions and a Gradient of Recognition , 1999 .

[49]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[50]  Daw-Tung Lin,et al.  Facial expressions classification with hierarchical radial basis function networks , 1999, ICONIP'99. ANZIIS'99 & ANNES'99 & ACNN'99. 6th International Conference on Neural Information Processing. Proceedings (Cat. No.99EX378).

[51]  J. N. Bassili.  Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face. , 1979, Journal of personality and social psychology.

[52]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[53]  James Jenn-Jier Lien,et al.  A Multi-Method Approach for Discriminating Between Similar Facial Expressions, Including Expression Intensity Estimation , 1998 .

[54]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[55]  Jesse Hoey,et al.  Hierarchical unsupervised learning of facial expression categories , 2001, Proceedings IEEE Workshop on Detection and Recognition of Events in Video.

[56]  P. Ekman.  Facial expressions of emotion: an old controversy and new findings. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[57]  P. Ekman.  Facial expression and emotion. , 1993, The American psychologist.

[58]  Rosalind W. Picard,et al.  Towards a Learning Companion that Recognizes Affect , 2001 .

[59]  Larry S. Davis,et al.  Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow , 1996, IEEE Trans. Pattern Anal. Mach. Intell..