Boosting Hankel matrices for face emotion recognition and pain detection

HighligthsDynamics of face expression descriptors are modeled for emotion recognition.A set of Hankel matrices is built upon several multi-scale face representations.Boosting and random subspace projection are used for dynamics selection.Dynamics of Haar-like features and Gabor Energies are compared.Fine-grained dynamics of subtle expressions can be modeled at small spatial scales. Studies in psychology have shown that the dynamics of emotional expressions play an important role in face emotion recognition in humans. Motivated by these studies, in this paper the dynamics of face expressions are modeled and used for automatic emotion recognition and pain detection.Given a temporal sequence of face images, several appearance-based descriptors are computed at each frame. Over the sequence, the descriptors corresponding to the same feature type and spatial scale define a time series. The Hankel matrix built upon each time series is used to represent the dynamics of face expressions with respect to the used feature-scale pair.The set of Hankel matrices obtained by varying the feature type and the scale is used within a boosting approach to train a strong classifier. During training, random subspace projection is adopted for feature and scale selection.Experiments on two challenging publicly available datasets show that the dynamics of appearance-based face expression representations can be used to discriminate among different emotion classes and, within a boosting approach, attain state-of-the-art average accuracy values in classification.