Adaptive facial expression recognition using inter-modal top-down context

The role of context in recognizing a person's affect is being increasingly studied. In particular, context arising from the presence of multi-modal information such as faces, speech and head pose has been used in recent studies to recognize facial expressions. In most approaches, the modalities are independently considered and the effect of one modality on the other, which we call inter-modal influence (e.g. speech or head pose modifying the facial appearance) is not modeled. In this paper, we describe a system that utilizes context from the presence of such inter-modal influences to recognize facial expressions. To do so, we use 2-D contextual masks which are activated within the facial expression recognition pipeline depending on the prevailing context. We also describe a framework called the Context Engine. The Context Engine offers a scalable mechanism for extending the current system to address additional modes of context that may arise during human-machine interactions. Results on standard data sets demonstrate the utility of modeling inter-modal contextual effects in recognizing facial expressions.

[1]  Michael C. Nechyba,et al.  PittPatt Face Detection and Tracking for the CLEAR 2006 Evaluation , 2006, CLEAR.

[2]  Loïc Kessous,et al.  Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech , 2008, Affect and Emotion in Human-Computer Interaction.

[3]  Björn W. Schuller,et al.  Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling , 2010, INTERSPEECH.

[4]  Thomas S. Huang,et al.  Emotion Recognition Based on Multimodal Information , 2009, Affective Information Processing.

[5]  Ravi Kiran Sarvadevabhatla,et al.  Panoramic attention for humanoid robots , 2009, 2009 9th IEEE-RAS International Conference on Humanoid Robots.

[6]  Marie L. Smith,et al.  Transmission of Facial Expressions of Emotion Co-Evolved with Their Efficient Decoding in the Brain: Behavioral and Brain Evidence , 2009, PloS one.

[7]  Carlos Busso,et al.  IEMOCAP: interactive emotional dyadic motion capture database , 2008, Lang. Resour. Evaluation.

[8]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Enrique Muñoz,et al.  Recognising facial expressions in video sequences , 2007, Pattern Analysis and Applications.

[10]  Peter Robinson,et al.  Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[11]  Vijay K. Madisetti,et al.  The Digital Signal Processing Handbook , 1997 .

[12]  Paul S. Bradley,et al.  Feature Selection via Concave Minimization and Support Vector Machines , 1998, ICML.

[13]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[14]  R. S. Jadon,et al.  Effectiveness of Eigenspaces for Facial Expressions Recognition , 2009 .

[15]  Aggelos K. Katsaggelos,et al.  Automatic facial expression recognition using facial animation parameters and multistream HMMs , 2006, IEEE Transactions on Information Forensics and Security.

[16]  Mohammed Yeasin,et al.  Recognition of facial expressions and measurement of levels of interest from video , 2006, IEEE Transactions on Multimedia.

[17]  Yaacov Trope,et al.  Putting Facial Expressions Back in Context , 2008 .

[18]  Richard Bowden,et al.  The Effect of Pose on Facial Expression Recognition , 2009, BMVC.

[19]  Matti Pietikäinen,et al.  Face Description with Local Binary Patterns: Application to Face Recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Y. Bar-Shalom.  Tracking and data association , 1988 .

[21]  Zhigang Deng,et al.  Analysis of emotion recognition using facial expressions, speech and multimodal information , 2004, ICMI '04.

[22]  Fadi Dornaika,et al.  Simultaneous Facial Action Tracking and Expression Recognition in the Presence of Head Motion , 2008, International Journal of Computer Vision.

[23]  L. F. Barrett,et al.  Context Is Routinely Encoded During Emotion Perception , 2010, Psychological science.

[24]  Rakesh Tripathi,et al.  Recognizing Facial Expression Using Particle Filter Based Feature Points Tracker , 2007, PReMI.

[25]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[26]  Nicu Sebe,et al.  MULTIMODAL EMOTION RECOGNITION , 2005 .

[27]  Xizhao Wang,et al.  Enhancing Generalization Capability of SVM Classifiers with Feature Weight Adjustment , 2004, KES.

[28]  A. Young,et al.  Configural information in facial expression perception. , 2000, Journal of experimental psychology. Human perception and performance.

[29]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[30]  Gary R. Bradski,et al.  Real time face and object tracking as a component of a perceptual user interface , 1998, Proceedings Fourth IEEE Workshop on Applications of Computer Vision. WACV'98 (Cat. No.98EX201).

[31]  Luc Van Gool,et al.  Hough Forest-Based Facial Expression Recognition from Video Sequences , 2010, ECCV Workshops.

[32]  Kostas Karpouzis,et al.  Robust Feature Detection for Facial Expression Recognition , 2007, EURASIP J. Image Video Process..

[33]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.