Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures

This paper presents a system for inferring complex mental states from video of facial expressions and head gestures in real-time. The system is based on a multi-level dynamic Bayesian network classifier which models complex mental states as a number of interacting facial and head displays, identified from component-based facial features. Experimental results for 6 mental states groups- agreement, concentrating, disagreement, interested, thinking and unsure are reported. Real-time performance, unobtrusiveness and lack of preprocessing make our system particularly suitable for user-independent human computer interaction.

[1]  Qiang Ji,et al.  Facial event classification with task oriented dynamic Bayesian network , 2004, CVPR 2004.

[2]  Jesse Hoey,et al.  Decision Theoretic Modeling of Human Facial Displays , 2004, ECCV.

[3]  Nicu Sebe,et al.  Learning Bayesian network classifiers for facial expression recognition both labeled and unlabeled data , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[4]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  S. Baron-Cohen.  Mindblindness: An Essay on Autism and Theory of Mind , 1997 .

[6]  Vladimir Pavlovic,et al.  Boosting and structure learning in dynamic Bayesian networks for audio-visual speaker detection , 2002, Object recognition supported by user interaction for service robots.

[7]  Takeo Kanade,et al.  Feature-point tracking by optical flow discriminates subtle differences in facial expression , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[8]  Simeon Keates,et al.  Temporal context and the recognition of emotion from facial expression , 2003 .

[9]  Jesse Hoey,et al.  Bayesian clustering of optical flow fields , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[10]  Marco La Cascia,et al.  Fast, Reliable Head Tracking under Varying Illumination: An Approach Based on Registration of Texture-Mapped 3D Models , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  S. Baron-Cohen.  How to build a baby that can read minds: Cognitive mechanisms in mindreading. , 1994 .

[12]  A. Isen,et al.  Positive affect and decision making. , 1993 .

[13]  Qiang Ji,et al.  Facial expression understanding in image sequences using dynamic and active visual information fusion , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[14]  Vladimir Pavlovic,et al.  Bayesian networks as ensemble of classifiers , 2002, Object recognition supported by user interaction for service robots.

[15]  A. Damasio.  Descartes' error: emotion, reason, and the human brain. avon books , 1994 .

[16]  Ashish Kapoor,et al.  A real-time head nod and shake detector , 2001, PUI '01.

[17]  Simon Baron-Cohen,et al.  Reading the mind in the face: A cross-cultural and developmental study , 1996 .

[18]  Larry S. Davis,et al.  Recognition of head gestures using hidden Markov models , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[19]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[20]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Jing Xiao,et al.  Robust full-motion recovery of head by dynamic templates and re-registration techniques , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[22]  Jake K. Aggarwal,et al.  Semantic-level Understanding of Human Actions and Interactions using Event Hierarchy , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[23]  J. Cohn,et al.  Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding. , 1999, Psychophysiology.

[24]  P. Rozin,et al.  High frequency of facial expressions corresponding to confusion, concentration, and worry in an analysis of naturally occurring facial expressions of Americans. , 2003, Emotion.

[25]  Marian Stewart Bartlett,et al.  Classifying Facial Actions , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Sean A. Spence,et al.  Descartes' Error: Emotion, Reason and the Human Brain , 1995 .

[27]  Sing Bing Kang,et al.  Emerging Topics in Computer Vision , 2004 .

[28]  Thomas Marill,et al.  On the effectiveness of receptors in recognition systems , 1963, IEEE Trans. Inf. Theory.

[29]  Simon Baron-Cohen,et al.  The Essential Difference: The Truth About the Male and Female Brain , 2003 .

[30]  S. Baron-Cohen,et al.  Mind Reading: The Interactive Guide to Emotions , 2003 .

[31]  Gwen Littlewort,et al.  Towards Social Robots: Automatic Evaluation of Human-robot Interaction by Face Detection and Expression Classification , 2003, NIPS.

[32]  Thomas S. Huang,et al.  Facial Expression Recognition from Video Sequences : Temporal and Static Modelling , 2002 .

[33]  Stan Sclaroff,et al.  Automatic detection of relevant head gestures in American Sign Language communication , 2002, Object recognition supported by user interaction for service robots.

[34]  R. Birdwhistell.  Kinesics and context , 1970 .

[35]  A. Damasio,et al.  Emotion, decision making and the orbitofrontal cortex. , 2000, Cerebral cortex.

[36]  Maja Pantic,et al.  Expert system for automatic analysis of facial expressions , 2000, Image Vis. Comput..

[37]  R. Birdwhistell.  Kinesics and Context: Essays on Body Motion Communication , 1971 .

[38]  William H. Hsu,et al.  A Survey of Algorithms for Real-Time Bayesian Network Inference , 2002 .

[39]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[40]  Takeo Kanade,et al.  Automated facial expression recognition based on FACS action units , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[41]  Margot J. Taylor,et al.  Early processing of the six basic facial emotional expressions. , 2003, Brain research. Cognitive brain research.

[42]  S. Leekam,et al.  The Childs Theory of Mind - Wellman, Hm , 1992 .

[43]  L. F. Barrett,et al.  Handbook of Emotions , 1993 .

[44]  Shinjiro Kawato,et al.  Real-time detection of nodding and head-shaking by directly detecting and tracking the "between-eyes" , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).