Expression Recognition from Visible Images with the Help of Thermal Images

Most facial expression recognition research focused on visible spectrum, which is sensitive to illumination changes. While thermal images, recording facial temperature distribution, are robust to light conditions. Therefore, expression recognition by visible and thermal image fusion is promising. However, in most cases, only visible images are available, since thermal cameras are much more expensive than visible cameras, which are popular in our daily life. Thus, in this paper, we propose a novel visible expression recognition approach by using thermal infrared data as privileged information, which is only available during training. First, active appearance model parameters and three defined head motion features are extracted from visible spectrum images, and several thermal statistical features are extracted from thermal infrared images. Second, feature selection is performed using the F-test statistic. Third, a new visible feature space is constructed using canonical correlation analysis under the help of thermal infrared images. After that, a support vector machine is adopted as the classifier on the constructed visible feature space. Experiments on the NVIE and Equinox database show the effectiveness of the proposed methods, and demonstrate that thermal infrared images' supplementary role for visible facial expression recognition.