Non-frontal view facial expression recognition based on ergodic hidden Markov model supervectors

Automatic facial expression recognition from non-frontal views is a challenging research topic which has recently started to attract the attention of the research community. In this paper, we propose a novel approach to tackling this problem based on the ergodic hidden Markov model (EHMM) supervector representation of facial images. First, the scale-invariant feature transform (SIFT) feature vectors are extracted from a dense grid of every facial images. Next, an EHMM is trained over all facial images in the training set and is referred to as the universal background model (UBM). The UBM is then maximum a posteriori adapted to each facial image in the training and test sets to produce the image-specific EHMMs. Based on these EHMMs, we derive a supervector representation of the facial images by means of an upper bound approximation of the Kullback-Leibler divergence rate between two EHMMs. Finally, facial expression recognition is performed in the linear discriminant subspace of the EHMM supervectors using the k-nearest-neighbor classification algorithm. Our experiments of recognizing six universal facial expressions over extensive multiview facial images with seven pan angles (−45° ∼ +45°) and five tilt angles (−30° ∼ +30°), which are synthesized from the BU-3DFE facial expression database, show promising results compared to the state of the arts recently reported.