Automatic Analysis of Facial Expressions: The State of the Art

Humans detect and interpret faces and facial expressions in a scene with little or no effort. Still, development of an automated system that accomplishes this task is rather difficult. There are several related problems: detection of an image segment as a face, extraction of the facial expression information, and classification of the expression (e.g., in emotion categories). A system that performs these operations accurately and in real time would form a big step in achieving a human-like interaction between man and machine. The paper surveys the past work in solving these problems. The capability of the human visual system with respect to these problems is discussed, too. It is meant to serve as an ultimate goal and a guide for determining recommendations for development of an automatic facial expression analyzer.

[1]  A. Mehrabian.  Communication without words , 1968 .

[2]  P. Ekman.  Unmasking The Face , 1975 .

[3]  H. Ellis.  Recognizing faces. , 1975, British journal of psychology.

[4]  D. Rutter,et al.  The Role of Visual Communication in Social Exchange , 1976 .

[5]  J. N. Bassili.  Facial motion in the perception of faces and of emotional expression. , 1978, Journal of experimental psychology. Human perception and performance.

[6]  J. N. Bassili.  Facial motion in the perception of faces and of emotional expression. , 1978 .

[7]  David J. Hand,et al.  Discrimination and Classification , 1982 .

[8]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[9]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[10]  P. Ekman.  Emotion in the human face , 1982 .

[11]  Bruce Bowe.  The Face of Emotion , 1985 .

[12]  F. W. Campbell,et al.  How Much of the Information Falling on the Retina Reaches the Visual Cortex and How Much is Stored in the Visual Memory , 1985 .

[13]  Kanter,et al.  Associative recall of memory without errors. , 1987, Physical review. A, General physics.

[14]  Alan L. Yuille,et al.  Feature extraction from faces using deformable templates , 1989, Proceedings CVPR '89: IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  C. von der Malsburg,et al.  Distortion invariant object recognition by matching hierarchically labeled graphs , 1989, International 1989 Joint Conference on Neural Networks.

[16]  Garrison W. Cottrell,et al.  EMPATH: Face, Emotion, and Gender Recognition Using Holons , 1990, NIPS.

[17]  Mubarak Shah,et al.  A fast algorithm for active contours , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[18]  C. Izard.  Facial expressions and the regulation of emotions. , 1990, Journal of personality and social psychology.

[19]  Fumio Hara,et al.  The recognition of basic facial expressions by neural network , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[20]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[21]  A. J. Fridlund.  Evolution and facial action in reflex, social motive, and paralanguage , 1991, Biological Psychology.

[22]  Ashok Samal,et al.  Minimum resolution for human face detection and identification , 1991, Electronic Imaging.

[23]  Kenji Mase,et al.  Recognition of Facial Expression from Optical Flow , 1991 .

[24]  Fumio Hara,et al.  Recognition of Six basic facial expression and their strength by neural network , 1992, [1992] Proceedings IEEE International Workshop on Robot and Human Communication.

[25]  Ashok Samal,et al.  Automatic recognition and analysis of human faces and facial expressions: a survey , 1992, Pattern Recognit..

[26]  V. Bruce.  What the human face tells the human mind: some challenges for the robot-human interface , 1992, [1992] Proceedings IEEE International Workshop on Robot and Human Communication.

[27]  Mubarak Shah,et al.  A Fast algorithm for active contours and curvature estimation , 1992, CVGIP Image Underst..

[28]  D. J. Myers,et al.  Image Feature Location in Multi-Resolution Images Using a Hierarchy of Multilayer Perceptrons , 1992 .

[29]  O. Nakamura,et al.  Description and synthesis of facial expression based on isodensity maps , 1992 .

[30]  Fumio Hara,et al.  Recognition of mixed facial expressions by neural network , 1992, [1992] Proceedings IEEE International Workshop on Robot and Human Communication.

[31]  Daniel Thalmann,et al.  Six-hundred indexed references on computer animation , 1992, Comput. Animat. Virtual Worlds.

[32]  Arcot Sowmya,et al.  Neural network approach to component versus holistic recognition of facial expressions in images , 1992, Other Conferences.

[33]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[34]  T. Takagi,et al.  Recognition of facial expressions using conceptual fuzzy sets , 1993, [Proceedings 1993] Second IEEE International Conference on Fuzzy Systems.

[35]  H. Yamada.  Visual Information for Categorizing Facial Expression of Emotions , 1993 .

[36]  Sati McKenzie,et al.  Machine Interpretation of Emotion: Design of a Memory-Based Expert System for Interpreting Facial Expressions in Terms of Signaled Emotions , 1993, Cogn. Sci..

[37]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[38]  D. Stuss,et al.  Cognitive neuroscience. , 1993, Current opinion in neurobiology.

[39]  Pertti Roivainen,et al.  3-D Motion Estimation in Model-Based Facial Image Coding , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Vicki Bruce.  What the human face tells the human mind: some challenges for the robot-human interface , 1993, Adv. Robotics.

[41]  Eero P. Simoncelli.  Distributed representation and analysis of visual motion , 1993 .

[42]  G. Kearney,et al.  Machine Interpretation of Emotion: Design of a Memory‐Based Expert System for Interpreting Facial Expressions in Terms of Signaled Emotions , 1993 .

[43]  Fumio Hara,et al.  Recognition of Mixed Facial Expressions by Neural Network. , 1993 .

[44]  Akikazu Takeuchi,et al.  Communicative facial displays as a new conversational modality , 1993, INTERCHI.

[45]  Demetri Terzopoulos,et al.  Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  Larry S. Davis,et al.  Computing spatio-temporal representations of human faces , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[47]  A. Anderson,et al.  The Effects of Visibility on Dialogue and Performance in a Cooperative Problem Solving Task , 1994 .

[48]  J. Russell.  Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. , 1994, Psychological bulletin.

[49]  Larry Davis,et al.  Recognizing facial expressions by spatio-temporal analysis , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[50]  P. Ekman,et al.  Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique. , 1994, Psychological bulletin.

[51]  Alex Pentland,et al.  View-based and modular eigenspaces for face recognition , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[52]  M. Rosenblum,et al.  Human emotion recognition from motion using a radial basis function network architecture , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[53]  Hiroshi Harashima,et al.  3-D Emotion Space for Interactive Communication , 1995, ICSC.

[54]  Michael J. Black,et al.  Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion , 1995, Proceedings of IEEE International Conference on Computer Vision.

[55]  Anca L. Ralescu,et al.  Some issues in fuzzy and linguistic modeling , 1995, Proceedings of 1995 IEEE International Conference on Fuzzy Systems..

[56]  Andrew Blake,et al.  Determining facial expressions in real time , 1995, Proceedings of IEEE International Conference on Computer Vision.

[57]  Shigeo Morishima,et al.  A modeling of facial expression and emotion for recognition and synthesis , 1995 .

[58]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[59]  Alexander H. Waibel,et al.  A real-time face tracker , 1996, Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV'96.

[60]  Haiyuan Wu,et al.  Face and facial feature extraction from color image , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[61]  J. Ohya,et al.  Recognition of facial expressions using HMM with continuous output probabilities , 1996, Proceedings 5th IEEE International Workshop on Robot and Human Communication. RO-MAN'96 TSUKUBA.

[62]  Ryohei Nakatsu,et al.  Life-like communication agent-emotion sensing character "MIC" and feeling session character "MUSE" , 1996, Proceedings of the Third IEEE International Conference on Multimedia Computing and Systems.

[63]  Fumio Hara,et al.  State-of-the art in component technology for an animated face robot-its component technology development for interactive communication with humans , 1996, Adv. Robotics.

[64]  Garrison W. Cottrell,et al.  Representing Face Images for Emotion Classification , 1996, NIPS.

[65]  Robert J. Holt,et al.  Determining articulated motion from perspective views: a decomposition approach , 1997, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[66]  Katsuhiko Shirai,et al.  Facial expressions recognition using discrete Hopfield neural networks , 1997, Proceedings of International Conference on Image Processing.

[67]  Chung-Lin Huang,et al.  Facial Expression Recognition Using Model-Based Feature Extraction and Action Parameters Classification , 1997, J. Vis. Commun. Image Represent..

[68]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[69]  J. Russell,et al.  The psychology of facial expression: Frontmatter , 1997 .

[70]  Vladimir Pavlovic,et al.  Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[71]  M. Yachida,et al.  Facial expression recognition and its degree estimation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[72]  F. Hara,et al.  Facial interaction between animated 3D face robot and human beings , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[73]  L. de Silva,et al.  Facial emotion recognition using multi-modal information , 1997, Proceedings of ICICS, 1997 International Conference on Information, Communications and Signal Processing. Theme: Trends in Information Systems Engineering and Wireless Multimedia Communications (Cat..

[74]  Yasuhiro Nakamura,et al.  Mathematical representation and image generation of human faces by metamorphosis , 1997 .

[75]  Hartmut Neven,et al.  Online facial expression recognition based on personalized galleries , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[76]  Tomaso A. Poggio,et al.  Example-Based Learning for View-Based Human Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[77]  Hong Yan,et al.  An Analytic-to-Holistic Approach for Face Recognition Based on a Single Frontal View , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[78]  Takeo Kanade,et al.  Automated facial expression recognition based on FACS action units , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[79]  Timothy F. Cootes,et al.  Face Recognition Using Active Appearance Models , 1998, ECCV.

[80]  Matthew Stone,et al.  An anthropometric face model using variational techniques , 1998, SIGGRAPH.

[81]  Prem Kalra,et al.  Face to virtual face , 1998, Proc. IEEE.

[82]  Takeo Kanade,et al.  Neural Network-Based Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[83]  Hartmut Neven,et al.  PersonSpotter-fast and robust system for human detection, tracking and recognition , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[84]  Takeo Kanade,et al.  Feature-point tracking by optical flow discriminates subtle differences in facial expression , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[85]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[86]  Tsutomu Miyasato,et al.  Multimodal human emotion/expression recognition , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[87]  R. Nakatsu,et al.  Toward the creation of a new medium for the multimedia era , 1998, Proc. IEEE.

[88]  Jun Ohya,et al.  Spotting segments displaying facial expression from image sequences using HMM , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[89]  Shigeru Akamatsu,et al.  Automatic detection of human faces in natural scene images by use of a skin color model and of invariant moments , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[90]  Prem Kalra,et al.  Direct Face-to-Face Communication Between Real and Virtual Humans , 1998 .

[91]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[92]  Peter Eisert,et al.  Analyzing Facial Expressions for Virtual Conferencing , 1998, IEEE Computer Graphics and Applications.

[93]  Masahiko Yachida,et al.  Expression recognition from time-sequential facial images by use of expression change model , 1997, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[94]  Zhengyou Zhang,et al.  Comparison between geometry-based and Gabor-wavelets-based facial expression recognition using multi-layer perceptron , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[95]  T. Sejnowski,et al.  Measuring facial expressions by computer image analysis. , 1999, Psychophysiology.

[96]  Michael J. Lyons,et al.  Automatic Classification of Single Facial Images , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[97]  Aaron F. Bobick,et al.  Parametric Hidden Markov Models for Gesture Recognition , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[98]  Jin-Hyung Kim,et al.  An HMM-Based Threshold Model Approach for Gesture Recognition , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[99]  Valery A. Petrushin,et al.  EMOTION IN SPEECH: RECOGNITION AND APPLICATION TO CALL CENTERS , 1999 .

[100]  Maja Pantic,et al.  An expert system for multiple emotional classification of facial expressions , 1999, Proceedings 11th International Conference on Tools with Artificial Intelligence.

[101]  Marian Stewart Bartlett,et al.  Classifying Facial Actions , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[102]  Maja Pantic,et al.  Expert system for automatic analysis of facial expressions , 2000, Image Vis. Comput..

[103]  William H. Press,et al.  Numerical recipes in C , 2002 .