Analysis of Machine Learning Methods for Real-Time Recognition of Facial Expressions from Video

We present a systematic comparison of machine learning methods applied to the problem of fully automatic recognition of facial expressions, including AdaBoost, support ve ctor machines, and linear discriminant analysis. Each video frame is first scanned in real-time to detect upright-fronta l faces. The faces found are scaled into image patches of equal size and sent downstream for further processing. Gabor energy filters are applied at the scaled image patches followed by a recognition engine that codes facial expressions into 7 dimensions in real time: neutral, anger, disgust, fear, joy, sadness, surprise. We report results on a series of experiments comparing spatial frequency ranges, feature selection techniques, and recognition engines. Be t results were obtained by selecting a subset of Gabor filters using AdaBoost and then training Support Vector Machines on the outputs of the filters selected by AdaBoost. The generalization performance to new subjects for a 7-way forced choice was 93% and 97% correct on two publicly available datasets, the best performance reported so far on these datasets. Surprisingly, registration of internal facial f eatures was not necessary, even though the face detector does not provide precisely registered images. The outputs of the classifier change smoothly as a function of time and thus can be used for unobtrusive motion capture. We developed an end-to-end system that provides facial expression codes at 24 frames per second and animates a computer generated character in real time.