TV-GAN: Generative Adversarial Network Based Thermal to Visible Face Recognition

This work tackles the face recognition task on images captured using thermal camera sensors which can operate in the non-light environment. While it can greatly increase the scope and benefits of the current security surveillance systems, performing such a task using thermal images is a challenging problem compared to face recognition task in the Visible Light Domain (VLD). This is partly due to the significantly smaller amount of thermal imagery data collected compared to the VLD data. Unfortunately, direct application of the existing very strong face recognition models trained using VLD data into the thermal imagery data will not produce a satisfactory performance. This is due to the existence of the domain gap between the thermal and VLD images. To this end, we propose a Thermal-to-Visible Generative Adversarial Network (TV-GAN) that is able to transform thermal face images into their corresponding VLD images whilst maintaining identity information which is sufficient enough for the existing VLD face recognition models to perform recognition. Some examples are presented in Figure 1. Unlike the previous methods, our proposed TV-GAN uses an explicit closed-set face recognition loss to regularize the discriminator network training. This information will then be conveyed into the generator network in the form of gradient loss. In the experiment, we show that by using this additional explicit regularization for the discriminator network, the TV-GAN is able to preserve more identity information when translating a thermal image of a person which is not seen before by the TV-GAN.

[1]  Jonghyun Choi,et al.  Thermal-to-visible face recognition using partial least squares. , 2015, Journal of the Optical Society of America. A, Optics, image science, and vision.

[2]  Brian C. Lovell,et al.  An Early Experience Toward Developing Computer Aided Diagnosis for Gram-Stained Smears Images , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[3]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Larry S. Davis,et al.  Thermal to visible face recognition , 2012, Defense + Commercial Sensing.

[5]  Rama Chellappa,et al.  Coupled dictionaries for thermal to visible face recognition , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[6]  Yu-Bin Yang,et al.  Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections , 2016, ArXiv.

[7]  Yu Qiao,et al.  A Discriminative Feature Learning Approach for Deep Face Recognition , 2016, ECCV.

[8]  Abhinav Gupta,et al.  Generative Image Modeling Using Style and Structure Adversarial Networks , 2016, ECCV.

[9]  Aly A. Farag,et al.  Face recognition in low resolution thermal images , 2013, Comput. Vis. Image Underst..

[10]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[11]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[12]  Xiaoming Liu,et al.  Disentangled Representation Learning GAN for Pose-Invariant Face Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Prudhvi Gurram,et al.  A Polarimetric Thermal Database for Face Recognition Research , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[14]  Anil K. Jain,et al.  Heterogeneous Face Recognition Using Kernel Prototype Similarities , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Dimitris N. Metaxas,et al.  StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[16]  Shuowen Hu,et al.  Improving cross-modal face recognition using polarimetric imaging. , 2015, Optics letters.

[17]  Vishal M. Patel,et al.  Generative adversarial network-based synthesis of visible faces from polarimetrie thermal faces , 2017, 2017 IEEE International Joint Conference on Biometrics (IJCB).

[18]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[19]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[20]  Arun Ross,et al.  Matching thermal to visible face images using hidden factor analysis in a cascaded subspace learning framework , 2016, Pattern Recognit. Lett..

[21]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Heesung Kwon,et al.  Estimation of visible spectrum faces from polarimetric thermal faces , 2016, 2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems (BTAS).

[23]  Marios Savvides,et al.  NIR-VIS heterogeneous face recognition via cross-spectral joint dictionary learning and reconstruction , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[24]  Guillermo Sapiro,et al.  Not Afraid of the Dark: NIR-VIS Face Recognition via Cross-Spectral Hallucination and Low-Rank Embedding , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Erik Learned-Miller,et al.  Labeled Faces in the Wild : Updates and New Reporting Procedures , 2014 .

[26]  Yoshua Bengio,et al.  Generative Adversarial Networks , 2014, ArXiv.

[27]  Chao Zhang,et al.  Hallucinating faces from thermal infrared images , 2008, 2008 15th IEEE International Conference on Image Processing.

[28]  Vishal M. Patel,et al.  Generative Adversarial Network-based Synthesis of Visible Faces from Polarimetric Thermal Faces , 2017 .

[29]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[31]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Dong Yi,et al.  Face Matching Between Near Infrared and Visible Light Images , 2007, ICB.

[33]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[34]  Alessia Saggese,et al.  HEp-2 staining pattern recognition at cell and specimen levels: Datasets, algorithms and results , 2016, Pattern Recognit. Lett..

[35]  Mario Vento,et al.  Classifying Anti-nuclear Antibodies HEp-2 Images: A Benchmarking Platform , 2014, 2014 22nd International Conference on Pattern Recognition.

[36]  Shengcai Liao,et al.  Learning Face Representation from Scratch , 2014, ArXiv.

[37]  M. Saquib Sarfraz,et al.  Deep Perceptual Mapping for Cross-Modal Face Recognition , 2016, International Journal of Computer Vision.

[38]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[39]  Alessia Saggese,et al.  Computer Aided Diagnosis for Anti-Nuclear Antibodies HEp-2 images: Progress and challenges , 2016, Pattern Recognit. Lett..