Coding, Analysis, Interpretation, and Recognition of Facial Expressions

We describe a computer vision system for observing facial motion by using an optimal estimation optical flow method coupled with geometric, physical and motion-based dynamic models describing the facial structure. Our method produces a reliable parametric representation of the face's independent muscle action groups, as well as an accurate estimate of facial motion. Previous efforts at analysis of facial expression have been based on the facial action coding system (FACS), a representation developed in order to allow human psychologists to code expression from static pictures. To avoid use of this heuristic coding scheme, we have used our computer vision system to probabilistically characterize facial motion and muscle activation in an experimental population, thus deriving a new, more accurate, representation of human facial expressions that we call FACS+. Finally, we show how this method can be used for coding, analysis, interpretation, and recognition of facial expressions.

[1]  Alex Pentland,et al.  Closed-form solutions for physically-based shape modeling and recognition , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[2]  Alex Pentland,et al.  Facial expression recognition using a dynamic model and motion energy , 1995, Proceedings of IEEE International Conference on Computer Vision.

[3]  K. Bathe.  Finite Element Procedures , 1995 .

[4]  Bernard Friedland,et al.  Control System Design: An Introduction to State-Space Methods , 1987 .

[5]  C. Izard.  Facial expressions and the regulation of emotions. , 1990, Journal of personality and social psychology.

[6]  Robert Grover Brown,et al.  Introduction to random signal analysis and Kalman filtering , 1983 .

[7]  Edward H. Adelson,et al.  Layered representation for motion analysis , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Steven D. Pieper,et al.  Interactive graphics for plastic surgery: a task-level analysis and implementation , 1992, I3D '92.

[9]  H. Saunders,et al.  Finite element procedures in engineering analysis , 1982 .

[10]  Alex Pentland,et al.  Face recognition using view-based and modular eigenspaces , 1994, Optics & Photonics.

[11]  Pertti Roivainen,et al.  3-D Motion Estimation in Model-Based Facial Image Coding , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Alex Pentland,et al.  Closed-Form Solutions for Physically Based Shape Modeling and Recognition , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Alex Pentland,et al.  View-based and modular eigenspaces for face recognition , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[14]  J. N. Bassili.  Facial motion in the perception of faces and of emotional expression. , 1978, Journal of experimental psychology. Human perception and performance.

[15]  Norman I. Badler,et al.  Animating facial expressions , 1981, SIGGRAPH '81.

[16]  Dimitris N. Metaxas,et al.  Shape and Nonrigid Motion Estimation Through Physics-Based Synthesis , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Alex Pentland,et al.  Modeling, tracking and interactive animation of faces and heads//using input from video , 1996, Proceedings Computer Animation '96.

[18]  Thomas S. Huang,et al.  Final Report To NSF of the Planning Workshop on Facial Expression Understanding , 1992 .

[19]  S. A. Wainwright,et al.  Mechanical Design in Organisms , 2020 .

[20]  Irfan Essa,et al.  Analysis, interpretation and synthesis of facial expressions , 1995 .

[21]  Alex Pentland,et al.  Automatic lipreading by optical-flow analysis , 1989 .

[22]  Norman I. Badler,et al.  Final Report to Nsf of the Standards for Facial Animation Workshop Final Report to Nsf of the Standards for Facial Animation Workshop , 1994 .

[23]  Michael J. Black,et al.  Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion , 1995, Proceedings of IEEE International Conference on Computer Vision.

[24]  Eero P. Simoncelli.  Distributed representation and analysis of visual motion , 1993 .

[25]  Kenji Mase,et al.  Recognition of Facial Expression from Optical Flow , 1991 .

[26]  Alex Pentland,et al.  A Unified Approach for Physical and Geometric Modeling for Graphics and Animation , 1992, Comput. Graph. Forum.

[27]  James A. Brunner,et al.  Marketing and Negotiating in the People's Republic of China: Perceptions of American Businessmen Who Attended the 1975 Canton Fair , 1977 .

[28]  M. Minsky.  The Society of Mind , 1986 .

[29]  P. Ekman.  The argument and evidence about universals in facial expressions of emotion. , 1989 .

[30]  J. Bruner,et al.  THE PERCEPTION OF PEOPLE , 1954 .

[31]  C. Darwin.  The Expression of the Emotions in Man and Animals , .

[32]  Demetri Terzopoulos,et al.  Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  J. N. Bassili.  Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face. , 1979, Journal of personality and social psychology.

[34]  Irfan Essa,et al.  Physically-based Modeling for Graphics and Vision , 1992 .

[35]  Irfan Essa,et al.  Tracking facial motion , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[36]  Demetri Terzopoulos,et al.  Modelling and animating faces using scanned data , 1991, Comput. Animat. Virtual Worlds.

[37]  M. Rosenblum,et al.  Human emotion recognition from motion using a radial basis function network architecture , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[38]  Larry S. Davis,et al.  Computing spatio-temporal representations of human faces , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.