Recognizing Facial Expressions in Image Sequences Using Local Parameterized Models of Image Motion

This paper explores the use of local parametrized models of image motion for recovering and recognizing the non-rigid and articulated motion of human faces. Parametric flow models (for example affine) are popular for estimating motion in rigid scenes. We observe that within local regions in space and time, such models not only accurately model non-rigid facial motions but also provide a concise description of the motion in terms of a small number of parameters. These parameters are intuitively related to the motion of facial features during facial expressions and we show how expressions such as anger, happiness, surprise, fear, disgust, and sadness can be recognized from the local parametric motions in the presence of significant head motion. The motion tracking and expression recognition approach performed with high accuracy in extensive laboratory experiments involving 40 subjects as well as in television and movie sequences.

[1]  Demetri Terzopoulos,et al.  Snakes: Active contour models , 2004, International Journal of Computer Vision.

[2]  J. N. Bassili.  Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face. , 1979, Journal of personality and social psychology.

[3]  Michael Isard,et al.  3D position, attitude and shape input using video tracking of hands and lips , 1994, SIGGRAPH.

[4]  Stuart Geman,et al.  Statistical methods for tomographic image reconstruction , 1987 .

[5]  M. Rosenblum,et al.  Human emotion recognition from motion using a radial basis function network architecture , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[6]  Alex Pentland,et al.  Visually Controlled Graphics , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  A. Young,et al.  Handbook of Research on Face Processing , 1989 .

[8]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[9]  John Law,et al.  Robust Statistics—The Approach Based on Influence Functions , 1986 .

[10]  Pertti Roivainen,et al.  3-D Motion Estimation in Model-Based Facial Image Coding , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Demetri Terzopoulos,et al.  Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Tomaso Poggio,et al.  Example Based Image Analysis and Synthesis , 1993 .

[13]  Alan L. Yuille,et al.  Feature extraction from faces using deformable templates , 2004, International Journal of Computer Vision.

[14]  Andrew Blake,et al.  Surface Orientation and Time to Contact from Image Divergence and Deformation , 1992, ECCV.

[15]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[16]  Xiaobo Li,et al.  Towards a system for automatic facial feature detection , 1993, Pattern Recognit..

[17]  Larry S. Davis,et al.  Labeling of human face components from range data , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Larry S. Davis,et al.  Computing spatio-temporal representations of human faces , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Alan L. Yuille,et al.  Deformable templates , 1993 .

[20]  Irfan Essa,et al.  Tracking facial motion , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[21]  Andrea J. van Doorn,et al.  Invariant Properties of the Motion Parallax Field due to the Movement of Rigid Bodies Relative to an Observer , 1975 .

[22]  P. Ekman.  Facial expressions of emotion: an old controversy and new findings. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[23]  A. Jepson,et al.  Estimating multiple independent motions in segmented images using parametric models with local deformations , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[24]  Michael J. Black,et al.  The robust estimation of multiple motions: Affine and piecewise smooth flow fields , 1993 .

[25]  Alex Pentland,et al.  Recursive estimation of structure and motion using relative orientation constraints , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Michael J. Black,et al.  A framework for the robust estimation of optical flow , 1993, 1993 (4th) International Conference on Computer Vision.

[27]  Gilad Adiv,et al.  Determining Three-Dimensional Motion and Structure from Optical Flow Generated by Several Moving Objects , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  P. Ekman.  Emotion in the human face , 1982 .

[29]  Alex Pentland,et al.  A vision system for observing and extracting facial action parameters , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Sebastian Toelg,et al.  Towards an Example-Based Image Compression Architecture for Video-Conferencing , 1994 .

[31]  P. Ekman.  Unmasking The Face , 1975 .

[32]  Werner A. Stahel,et al.  Robust Statistics: The Approach Based on Influence Functions , 1987 .

[33]  Kenji Mase,et al.  Recognition of Facial Expression from Optical Flow , 1991 .