Temporal Facial Expression Modeling for Automated Action Unit Intensity Measurement

Spontaneous facial expression recognition using temporal patterns is a relatively unexplored area in facial image analysis. Several factors such as head orientation, co-occurrence and presence of subtle facial action units (AUs), and time variability of AUs make the problem more challenging. This paper presents a methodology to model and automatically recognize the intensity of spontaneous AUs in videos. Our method exploits localized Gabor features and Hidden Markov Model (HMM) to represent and model the dependencies of AU dynamics in both subject-dependent (SD) and subject-independent (SI) settings. Our experimental results show that temporal information can improve the recognition of AUs and their intensity levels compared to static methods.

[1]  Anibal Gutierrez,et al.  Children-Robot Interaction: Eye Gaze Analysis of Children with Autism During Social Interactions , 2014 .

[2]  Friedhelm Schwenker,et al.  A Hidden Markov Model Based Approach for Facial Expression Recognition in Image Sequences , 2010, ANNPR.

[3]  Takeo Kanade,et al.  Detection, tracking, and classification of action units in facial expression , 2000, Robotics Auton. Syst..

[4]  Daniel S. Messinger,et al.  A framework for automated measurement of the intensity of non-posed Facial Action Units , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[5]  Qiang Ji,et al.  A Unified Probabilistic Framework for Spontaneous Facial Action Modeling and Understanding , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  R. Gur,et al.  Automated Facial Action Coding System for dynamic analysis of facial expressions in neuropsychiatric disorders , 2011, Journal of Neuroscience Methods.

[7]  Qiang Ji,et al.  A unified probabilistic framework for measuring the intensity of spontaneous facial action units , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[8]  Anibal Gutierrez,et al.  Comparing the gaze responses of children with autism and typically developed individuals in human-robot interaction , 2014, 2014 IEEE-RAS International Conference on Humanoid Robots.

[9]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[10]  Stefanos Zafeiriou,et al.  Markov Random Field Structures for Facial Action Unit Intensity Estimation , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[11]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[12]  Junji Yamato,et al.  Recognizing human action in time-sequential images using hidden Markov model , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[13]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[14]  Nicu Sebe,et al.  Facial expression recognition from video sequences: temporal and static modeling , 2003, Comput. Vis. Image Underst..

[15]  Mohammad H. Mahoor,et al.  Facial action unit recognition with sparse representation , 2011, Face and Gesture 2011.

[16]  Gwen Littlewort,et al.  Dynamics of Facial Expression Extracted Automatically from Video , 2004, CVPR Workshops.

[17]  Mohammad H. Mahoor,et al.  Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses , 2014, Image Vis. Comput..

[18]  Mohammad H. Mahoor,et al.  Manifold alignment using curvature information , 2013, 2013 28th International Conference on Image and Vision Computing New Zealand (IVCNZ 2013).

[19]  Katherine B. Martin,et al.  Facial Action Coding System , 2015 .

[20]  Mohammad H. Mahoor,et al.  Automatic detection of non-posed facial action units , 2012, 2012 19th IEEE International Conference on Image Processing.

[21]  Fernando De la Torre,et al.  Temporal Segmentation of Facial Behavior , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[22]  Biing-Hwang Juang,et al.  Hidden Markov Models for Speech Recognition , 1991 .

[23]  Marian Stewart Bartlett,et al.  Automatic facial expression recognition for intelligent tutoring systems , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[24]  Mohammad H. Mahoor,et al.  DISFA: A Spontaneous Facial Action Intensity Database , 2013, IEEE Transactions on Affective Computing.