Fusing multi-stream deep neural networks for facial expression recognition

Among the factors contributing to conveying emotional state of an individual is facial expression. It represents the most important nonverbal communication and a challenging task in the field of computer vision. In this work, we propose a combined deep architecture model for facial expression recognition that uses appearance and geometric features extracted separately using convolution layers and supervised decent method, respectively. The proposed model is trained on three public databases [the Extended Cohn Kanade (CK+), the OULU-CASIA VIS, and the JAFFE]. The three databases contain a limited amount of data that we enlarge by adding a step of data augmentation to original images. For further comparison, two additional models that use appearance features only and geometric features only are trained on the same subset of data, to show how the combination of the two deep architectures influences results. On the other hand, in order to investigate the generalization of the combined model, a cross-database evaluation is performed. The obtained results achieve the state-of-the-art and improve recent work, especially in case of cross-database evaluation.