Automatic recognition of eye blinking in spontaneously occurring behavior

Previous researchin automatic facial expression recognition has been limited to recognition of gross expression categories (e.g., joy or anger) in posed facial behavior under well-controlled conditions (e.g., frontal pose and minimal out-of-plane head motion). We have developed a system that detects a discrete and important facial action (e.g., eye blinking) in spontaneously occurring facial behavior that has been measured with a nonfrontal pose, moderate out-of-plane head motion, and occlusion. The system recovers three-dimensional motion parameters, stabilizes facial regions, extracts motion and appearance information, and recognizes discrete facial actions in spontaneous facial behavior. We tested the system in video data from a two-person interview. The 10 subjects were ethnically diverse, action units occurred during speech, and out-of-plane motion and occlusion from head motion and glasses were common. The video data were originally collected to answer substantive questions in psychology and represent a substantial challenge to automated action unit recognition. In analysis of blinks, the system achieved 98% accuracy.

[1]  E. C. Grant,et al.  Human Facial Expression , 1969 .

[2]  M. Holland,et al.  Blinking and Mental Load , 1972, Psychological reports.

[3]  R J Harmon,et al.  Emotional expression in infancy; a biobehavioral study. , 1976, Psychological issues.

[4]  R. Kraut,et al.  Social and emotional messages of smiling: An ethological approach. , 1979 .

[5]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[6]  Bernd Neumann,et al.  Optical flow , 1986, Workshop on Motion.

[7]  W. Rinn,et al.  The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions. , 1984, Psychological bulletin.

[8]  N. Fox,et al.  Patterns of brain electrical activity during facial signs of emotion in 10-month-old infants. , 1988 .

[9]  Jeffrey F. Cohn,et al.  Effect of contingent changes in mothers' affective expression on the organization of behavior in 3-month-old infants , 1988 .

[10]  M. Katsikitis,et al.  A study of facial expression in Parkinson's disease using a novel microcomputer-based method. , 1988, Journal of neurology, neurosurgery, and psychiatry.

[11]  Karson Cn.  Physiology of normal and abnormal blinking. , 1988 .

[12]  R. Matias,et al.  A Comparison of Two Systems that Code Infant Affective Expression. , 1989 .

[13]  P. Ekman,et al.  The Duchenne smile: emotional expression and brain physiology. II. , 1990, Journal of personality and social psychology.

[14]  K. Craig,et al.  Genuine, suppressed and faked facial behavior during exacerbation of chronic low back pain , 1991, Pain.

[15]  Michael J. Black.  Robust incremental optical flow , 1992 .

[16]  George F. Michel,et al.  Infant interest expressions as coordinative motor structures , 1992 .

[17]  Bennett I. Bertenthal,et al.  Early Experience and Emotional Development: The Emergence of Wariness of Heights , 1992 .

[18]  P. Ekman.  Facial expression and emotion. , 1993, The American psychologist.

[19]  A. J. Fridlund.  Human Facial Expression: An Evolutionary View , 1994 .

[20]  Larry S. Davis,et al.  Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Dimitris N. Metaxas,et al.  The integration of optical flow and deformable models with applications to human face shape and motion estimation , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[22]  L. Camras,et al.  Infant “surprise” expressions as coordinative motor structures , 1996 .

[23]  P. Ekman,et al.  The ability to detect deceit generalizes across different types of high-stake lies. , 1997, Journal of personality and social psychology.

[24]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  J. Cohn,et al.  Mothers' affective behavior with infant siblings: stability and change. , 1997, Developmental psychology.

[26]  A. Young,et al.  In the Eye of the Beholder: The Science of Face Perception , 1998 .

[27]  Takeo Kanade,et al.  Neural Network-Based Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  T. Sejnowski,et al.  Measuring facial expressions by computer image analysis. , 1999, Psychophysiology.

[29]  J. Cohn,et al.  Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding. , 1999, Psychophysiology.

[30]  J. Cohn,et al.  Specific Impairment of Smiling Increases the Severity of Depressive Symptoms in Patients with Facial Neuromuscular Disorders , 1999, Aesthetic Plastic Surgery.

[31]  Marian Stewart Bartlett,et al.  Classifying Facial Actions , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Takeo Kanade,et al.  Eye-State Action Unit Detection by Gabor Wavelets , 2000, ICMI.

[33]  Takeo Kanade,et al.  Detection, tracking, and classification of action units in facial expression , 2000, Robotics Auton. Syst..

[34]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[35]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Karen L. Schmidt,et al.  Human facial expressions as adaptations: Evolutionary questions in facial expression research. , 2001, American journal of physical anthropology.

[37]  Automated face analysis (afa) distinguishes deliberate from spontaneous smiles on the basis of quantitative differences in intensity and asymmetry , 2001 .

[38]  J. Cohn,et al.  A Psychometric Evaluation of the Facial Action Coding System for Assessing Spontaneous Expression , 2001 .

[39]  Takeo Kanade,et al.  A Comparative Study of Alternative FACS Coding Algorithms , 2001 .

[40]  Yanxi Liu,et al.  Facial asymmetry quantification for expression invariant human identification , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[41]  Jing Xiao,et al.  Robust full-motion recovery of head by dynamic templates and re-registration techniques , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[42]  Ralph Gross,et al.  Individual differences in facial expression: stability over time, relation to self-reported emotion, and ability to inform person identification , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[43]  Takeo Kanade,et al.  Evaluation of Gabor-wavelet-based facial action unit recognition in image sequences of increasing complexity , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[44]  Karen L. Schmidt,et al.  Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles , 2003, Biological Psychology.

[45]  Jeffrey F. Cohn,et al.  The Timing of Facial Motion in posed and Spontaneous Smiles , 2003, Int. J. Wavelets Multiresolution Inf. Process..

[46]  Jing Xiao,et al.  Robust full‐motion recovery of head by dynamic templates and re‐registration techniques , 2003 .