Automatic Analysis of Spontaneous Facial Behavior: A Final Project Report

The Facial Action Coding System (FACS) is the leading standard for measuring facial expressions in the behavioral sciences (Ekman & Friesen, 1978). FACS coding is presently performed manually by human experts, it is slow, and requires extensive training. Automating FACS coding could have revolutionary effects in our understanding of human facial expression and on the development of computer systems that understand facial expressions. Two teams, one at University of California San Diego and the Salk Institute, and another at University of Pittsburgh and Carnegie Mellon University, were challenged to develop prototype systems for automatic recognition of spontaneous facial expressions. Working with spontaneous expressions required solving technical and theoretical challenges which had not been previously addressed in the field. This document describes the system developed by the UCSD team. The approach employs 3-D pose estimation and warping techniques to reduce image variability due to general changes in pose. Machine learning techniques are then applied directly on the warped images or on biologically inspired representations of these images. No efforts are made to detect contours or other hand-crafted image features. This system employed general purpose learning mechanisms that can be applied to recognition of any action unit. The approach is parsimonious and does not require defining a different set of feature parameters or image operations for each facial action. The system was tested on a set of eyelid and eyebrow movements and successfully identified these movements in novel subjects. We showed that 3D tracking and warping followed by machine learning techniques directly applied to the warped images, is a viable and promising technology for automatic facial action recognition. One exciting aspect of the approach presented here is that information about movement dynamics emerged out of filters which were derived from the statistics of images. We believe all the pieces of the puzzle are ready for the development of automated systems that recognize spontaneous facial actions at the level of detail required by FACS. The main factor impeding development in this field is the lack of sufficiently large databases for training a greater variety of action units, and which may become a standard for comparison between different approaches. Based on our experience in this project we estimate that a database of 500 subjects, with 1 minute of rich facial behavior per subject, would be sufficient for dramatic improvements in the field.