LAFTER: a real-time face and lips tracker with facial expression recognition

This paper describes an active-camera real-time system for tracking, shape description, and classi"cation of the human face and mouth expressions using only a PC or equivalent computer. The system is based on use of 2-D blob features, which are spatially compact clusters of pixels that are similar in terms of low-level image properties. Patterns of behavior (e.g., facial expressions and head movements) can be classi"ed in real-time using hidden Markov models (HMMs). The system has been tested on hundreds of users and has demonstrated extremely reliable and accurate performance. Typical facial expression classi"cation accuracies are near 100%. ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved.

[1]  Timothy F. Cootes,et al.  A unified approach to coding and interpreting face images , 1995, Proceedings of IEEE International Conference on Computer Vision.

[2]  Alex Pentland.  Classification by Clustering , 1976 .

[3]  Curtis Padgett,et al.  Representing F Ace Images for Emotion Classiication , 2022 .

[4]  Chil-Woo Lee,et al.  Automatic recognition of human facial expressions , 1995, Proceedings of IEEE International Conference on Computer Vision.

[5]  R. E. Kalman,et al.  New Results in Linear Filtering and Prediction Theory , 1961 .

[6]  Robert C. Bolles,et al.  The Representation Space Paradigm of Concurrent Evolving Object Descriptions , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Alex Pentland,et al.  Camera Self-Calibration From One Point Correspondence , 1995 .

[8]  K. Prasad,et al.  Using Deformable Templates to Infer Visual Speech , 1994 .

[9]  Alex Pentland,et al.  Pfinder: real-time tracking of the human body , 1996, Other Conferences.

[10]  Stephen M. Omohundro,et al.  Surface Learning with Applications to Lipreading , 1993, NIPS.

[11]  David G. Stork,et al.  Using deformable templates to infer visual speech dynamics , 1994, Proceedings of 1994 28th Asilomar Conference on Signals, Systems and Computers.

[12]  A. Pentland,et al.  Blob - An unsupervised clustering approach to spatial preprocessing of MSS imagery , 1977 .

[13]  William W. Gaver.  The affordances of media spaces for collaboration , 1992, CSCW '92.

[14]  Junji Yamato,et al.  Recognizing human action in time-sequential images using hidden Markov model , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  R. B. Macleod,et al.  A Source Book Of Gestalt Psychology , 1939 .

[16]  J. Gibson.  The Ecological Approach to Visual Perception , 1979 .

[17]  Alex Pentland,et al.  Parametrized structure from motion for 3D adaptive feedback tracking of faces , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[18]  Keith Waters,et al.  A muscle model for animation three-dimensional facial expression , 1987, SIGGRAPH.

[19]  Alex Pentland,et al.  Facial expression recognition using a dynamic model and motion energy , 1995, Proceedings of IEEE International Conference on Computer Vision.

[20]  Thad Starner,et al.  Visual Recognition of American Sign Language Using Hidden Markov Models. , 1995 .

[21]  Alex Waibel,et al.  Face locating and tracking for human-computer interaction , 1994, Proceedings of 1994 28th Asilomar Conference on Signals, Systems and Computers.

[22]  I. Pilowsky,et al.  Towards the quantification of facial expressions with the use of a mathematic model of the face , 1986 .

[23]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[24]  Aaron F. Bobick,et al.  Recognition and interpretation of parametric gesture , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[25]  Alex Waibel,et al.  Gaze Tracking Based on Face‐Color , 1995 .

[26]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[27]  Alex Pentland,et al.  Segmentation by minimal description , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[28]  Harpreet S. Sawhney,et al.  Layered representation of motion video using robust maximum-likelihood estimation of mixture models and MDL encoding , 1995, Proceedings of IEEE International Conference on Computer Vision.

[29]  Alex Pentland,et al.  Pfinder: real-time tracking of the human body , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[30]  B. Ripley,et al.  Pattern Recognition , 1968, Nature.

[31]  H. Martin Hunke,et al.  Locating and Tracking of Human Faces with Neural Networks , 1994 .

[32]  A. Young,et al.  Aspects of face processing , 1986 .

[33]  W. Andrew.  LO, . Finance: Survey.. Journal of the American Statistical Association, , . , 2000 .

[34]  James L. Crowley,et al.  Multi-modal tracking of faces for video communications , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[35]  Michael J. Black,et al.  Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion , 1995, Proceedings of IEEE International Conference on Computer Vision.

[36]  Alexandros Eleftheriadis,et al.  Model-assisted coding of video teleconferencing sequences at low bit rates , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[37]  H. Yamada,et al.  Dimensions of visual information for categorizing facial expressions of emotion , 1993 .

[38]  Irfan Essa,et al.  Analysis, interpretation and synthesis of facial expressions , 1995 .

[39]  William W. Gaver,et al.  A Virtual Window on media space , 1995, CHI '95.

[40]  Larry S. Davis,et al.  Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  Aaron F. Bobick,et al.  Learning visual behavior for gesture analysis , 1995, Proceedings of International Symposium on Computer Vision - ISCV.

[42]  Kazuo Ohzeki,et al.  Interactive model-based coding of facial image sequence with a new motion detection algorithm , 1996 .

[43]  D. Titterington.  Recursive Parameter Estimation Using Incomplete Data , 1984 .

[44]  Andrew Blake,et al.  Determining facial expressions in real time , 1995, Proceedings of IEEE International Conference on Computer Vision.

[45]  Alex Pentland,et al.  LAFTER: lips and face real time tracker , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[46]  M. Wertheimer.  A source book of Gestalt psychology. , 1939 .

[47]  R. E. Kalman,et al.  A New Approach to Linear Filtering and Prediction Problems , 2002 .

[48]  Alex Pentland,et al.  Cooperative Robust Estimation Using Layers of Support , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[49]  Kiyoharu Aizawa,et al.  Model-based image coding advanced video coding techniques for very low bit-rate applications , 1995, Proc. IEEE.

[50]  R. Redner,et al.  Mixture densities, maximum likelihood, and the EM algorithm , 1984 .

[51]  Garrison W. Cottrell,et al.  Representing Face Images for Emotion Classification , 1996, NIPS.