Classifying Facial Actions

The Facial Action Coding System (FACS) [23] is an objective method for quantifying facial movement in terms of component actions. This system is widely used in behavioral investigations of emotion, cognitive processes, and social interaction. The coding is presently performed by highly trained human experts. This paper explores and compares techniques for automatically recognizing facial actions in sequences of images. These techniques include analysis of facial motion through estimation of optical flow; holistic spatial analysis, such as principal component analysis, independent component analysis, local feature analysis, and linear discriminant analysis; and methods based on the outputs of local filters, such as Gabor wavelet representations and local principal components. Performance of these systems is compared to naive and expert human subjects. Best performances were obtained using the Gabor wavelet representation and the independent component representation, both of which achieved 96 percent accuracy for classifying 12 facial actions of the upper and lower face. The results provide converging evidence for the importance of using local filters, high spatial frequencies, and statistical independence for classifying facial actions.

[1]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[2]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[3]  P. O. Bishop,et al.  Spatial vision. , 1971, Annual review of psychology.

[4]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[5]  J. N. Bassili.  Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face. , 1979, Journal of personality and social psychology.

[6]  D. Pollen,et al.  Phase relationships between adjacent simple cells in the visual cortex. , 1981, Science.

[7]  P. Ekman.  Telling lies: clues to deceit in the marketplace , 1985 .

[8]  J. P. Jones,et al.  An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. , 1987, Journal of neurophysiology.

[9]  John G. Daugman,et al.  Complete discrete 2-D Gabor transforms by neural networks for image analysis and compression , 1988, IEEE Trans. Acoust. Speech Signal Process..

[10]  P. Ekman,et al.  Smiles when lying. , 1988, Journal of personality and social psychology.

[11]  Garrison W. Cottrell,et al.  EMPATH: Face, Emotion, and Gender Recognition Using Holons , 1990, NIPS.

[12]  Terrence J. Sejnowski,et al.  SEXNET: A Neural Network Identifies Sex From Human Faces , 1990, NIPS.

[13]  Michael S. Landy,et al.  Nonlinear Model of Neural Responses in Cat Visual Cortex , 1991 .

[14]  K. Craig,et al.  Genuine, suppressed and faked facial behavior during exacerbation of chronic low back pain , 1991, Pain.

[15]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[16]  Kenji Mase,et al.  Recognition of Facial Expression from Optical Flow , 1991 .

[17]  D. Heeger.  Nonlinear model of neural responses in cat visual cortex. , 1991 .

[18]  H. Wallbott.  Effects of distortion of spatial and temporal resolution of video stimuli on emotion attributions , 1992 .

[19]  A. Shashua.  Geometry and Photometry in 3D Visual Recognition , 1992 .

[20]  Joseph J. Atick,et al.  What Does the Retina Know about Natural Scenes? , 1992, Neural Computation.

[21]  S. Kaiser,et al.  Automated coding of facial behavior in human-computer interactions with facs , 1992 .

[22]  S. Ullman,et al.  Geometry and photometry in three-dimensional visual recognition , 1993 .

[23]  Roberto Brunelli,et al.  Face Recognition: Features Versus Templates , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Joachim M. Buhmann,et al.  Distortion Invariant Object Recognition in the Dynamic Link Architecture , 1993, IEEE Trans. Computers.

[25]  Tomaso Poggio,et al.  Example Based Image Analysis and Synthesis , 1993 .

[26]  D. Stuss,et al.  Cognitive neuroscience. , 1993, Current opinion in neurobiology.

[27]  Pertti Roivainen,et al.  3-D Motion Estimation in Model-Based Facial Image Coding , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Demetri Terzopoulos,et al.  Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  J. Nadal.  Non linear neurons in the low noise limit : a factorial code maximizes information transferJean , 1994 .

[30]  J. Nadal,et al.  Nonlinear neurons in the low-noise limit: a factorial code maximizes information transfer Network 5 , 1994 .

[31]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[32]  Javier R. Movellan,et al.  Visual Speech Recognition with Stochastic Networks , 1994, NIPS.

[33]  David Beymer,et al.  Face recognition under varying pose , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[34]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[35]  Peter W. Hallinan,et al.  A deformable model for the recognition of human faces under arbitrary illumination , 1995 .

[36]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[37]  S. McKelvie.  Emotional expression in upside-down faces: evidence for configurational and componential processing. , 1995, The British journal of social psychology.

[38]  Marian Stewart Bartlett,et al.  Classifying Facial Action , 1995, NIPS.

[39]  Larry S. Davis,et al.  Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Penio S. Penev,et al.  Local feature analysis: A general statistical theory for object representation , 1996 .

[41]  Michael S. Gray i.  A comparison of local versus global image decompositions for visual speechreading , 1996 .

[42]  Tomaso Poggio,et al.  Image Representations for Visual Learning , 1996, Science.

[43]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[44]  Larry S. Davis,et al.  Human expression recognition from motion using a radial basis function network architecture , 1996, IEEE Trans. Neural Networks.

[45]  Garrison W. Cottrell,et al.  Representing Face Images for Emotion Classification , 1996, NIPS.

[46]  Marian Stewart Bartlett,et al.  Viewpoint Invariant Face Recognition using Independent Component Analysis and Attractor Networks , 1996, NIPS.

[47]  D. Perrett,et al.  A specific neural substrate for perceiving facial expressions of disgust , 1997, Nature.

[48]  Eero P. Simoncelli.  Statistical models for images: compression, restoration and synthesis , 1997, Conference Record of the Thirty-First Asilomar Conference on Signals, Systems and Computers (Cat. No.97CB36136).

[49]  Tomaso A. Poggio,et al.  Linear Object Classes and Image Synthesis From a Single Example Image , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[50]  Timothy F. Cootes,et al.  Automatic Interpretation and Coding of Face Images Using Flexible Models , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[51]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  Bruno A. Olshausen,et al.  Inferring Sparse, Overcomplete Image Codes Using an Efficient Coding Framework , 1998, NIPS.

[53]  S. Kanfer.  Serious Business: The Art And Commerce Of Animation In America From Betty Boop To Toy Story , 1997 .

[54]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[55]  Nikolaus F. Troje,et al.  Separation of texture and shape in images of faces for image coding and synthesis , 1997 .

[56]  Jun Zhang,et al.  Pace recognition: eigenface, elastic matching, and neural nets , 1997, Proc. IEEE.

[57]  James Jenn-Jier Lien,et al.  A Multi-Method Approach for Discriminating Between Similar Facial Expressions, Including Expression Intensity Estimation , 1998 .

[58]  H. Damasio,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence: Special Issue on Perceptual Organization in Computer Vision , 1998 .

[59]  Marian Stewart Bartlett,et al.  Independent component representations for face recognition , 1998, Electronic Imaging.

[60]  M. Bartlett,et al.  Face image analysis by unsupervised learning and redundancy reduction , 1998 .

[61]  Takeo Kanade,et al.  Subtly different facial expression recognition and expression intensity estimation , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[62]  Harry Wechsler,et al.  The FERET database and evaluation procedure for face-recognition algorithms , 1998, Image Vis. Comput..

[63]  T. Sejnowski,et al.  Measuring facial expressions by computer image analysis. , 1999, Psychophysiology.

[64]  J. Cohn,et al.  Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding. , 1999, Psychophysiology.

[65]  Zhengyou Zhang,et al.  Feature-Based Facial Expression Recognition: Sensitivity Analysis and Experiments with A Multilayer Perceptron , 1999, Int. J. Pattern Recognit. Artif. Intell..

[66]  Vicki Bruce,et al.  Face Recognition: From Theory to Applications , 1999 .

[67]  Thomas de Quincey.  [C] , 2000, The Works of Thomas De Quincey, Vol. 1: Writings, 1799–1820.