Mutual Component Convolutional Neural Networks for Heterogeneous Face Recognition

Heterogeneous face recognition (HFR) aims to identify a person from different facial modalities, such as visible and near-infrared images. The main challenges of HFR lie in the large modality discrepancy and insufficient training samples. In this paper, we propose the mutual component convolutional neural network (MC-CNN), a modal-invariant deep learning framework, to tackle these two issues simultaneously. Our MC-CNN incorporates a generative module, i.e., the mutual component analysis (MCA), into modern deep CNNs by viewing MCA as a special fully connected (FC) layer. Based on deep features, this FC layer is designed to extract modal-independent hidden factors and is updated according to maximum likelihood analytic formulation instead of back propagation which prevents overfitting from limited data naturally. In addition, we develop an MCA loss to update the network for modal-invariant feature learning. Extensive experiments show that our MC-CNN outperforms several fine-tuned baseline models significantly. Our methods achieve the state-of-the-art performance on the CASIA NIR-VIS 2.0, CUHK NIR-VIS, and IIIT-D Sketch datasets.