A weighted feature extraction method based on temporal accumulation of optical flow for micro-expression recognition

Abstract Spatiotemporal features are widely used in micro-expression (ME) recognition to represent facial appearance and action. The features extracted from different face regions are usually given different weights according to the motion intensities in the corresponding regions. The weighted features are reported to be more discriminative than the unweighted ones for ME recognition. However, MEs are so subtle that their motion intensities are usually as low as noises, therefore small image noises can cause similar weights with MEs and degenerate the effectiveness of these weights. To address this issue, a novel weighted feature extraction method is proposed in this paper, whereby the neighboring optical flows in a time interval are accumulated to compute motion intensities. In this manner, the displacements caused by image noises in optical flow are decreased because these displacements are random and direction-inconsistent. Meanwhile, the displacements caused by facial expressions are enhanced because the displacements caused by facial expressions are usually direction-consistent among neighboring frames. The weights computed from the accumulated optical flows are multiplied with the spatiotemporal features, then the weighted features are fed to SVM to classify MEs. The experimental results demonstrate that our method achieves comparable recognition performances with the state-of-the-art methods on SMIC-HS and outperforms the state-of-the-art methods on CASME II.