From Eyes to Face Synthesis: a New Approach for Human-Centered Smart Surveillance

With the popularity of surveillance cameras and the development of deep learning, significant progress has been made in the field of smart surveillance. Face recognition is one of the most important yet challenging tasks in human-centered smart surveillance, especially in public security, criminal investigation and anti-terrorism, and so on. Although, the state-of-the-art algorithms for face recognition have achieved dramatically improved results and have been widely applied in authentication scenario, the occlusion problem on face is still one of the critical issues for personal identification in smart surveillance, especially in the occasion of terrorist searching and identification. To address this issue, this paper proposed a new approach for eyes-to-face synthesis and personal identification for human-centered smart surveillance. An end-to-end network based on conditional generative adversarial networks (GAN) is designed to generate the face information based only on the available data of eyes region. To obtain photorealistic faces and identity-preserving information, a synthesis loss function based on feature loss, GAN loss, and total variation loss is proposed to guide the training process. Both the subject and objective experimental results demonstrated that the proposed method can preserve the identity based on eyes-only data, and provide a potential solution for the identification of person even in the case of face occlusion.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[3]  Minh N. Do,et al.  Semantic Image Inpainting with Deep Generative Models , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Ran He,et al.  Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[5]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  David Berthelot,et al.  BEGAN: Boundary Equilibrium Generative Adversarial Networks , 2017, ArXiv.

[7]  Yoshua Bengio,et al.  Better Mixing via Deep Representations , 2012, ICML.

[8]  Hao Li,et al.  High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Yu-Feng Yu,et al.  Discriminative multi-scale sparse coding for single-sample face recognition with occlusion , 2017, Pattern Recognit..

[10]  Rama Chellappa,et al.  Detection and Inpainting of Facial Wrinkles Using Texture Orientation Fields and Markov Random Field Modeling , 2014, IEEE Transactions on Image Processing.

[11]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  John P. Lewis,et al.  Face Inpainting with Local Linear Representations , 2004, BMVC.

[13]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[14]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Chokri Ben Amar,et al.  Video surveillance system based on a scalable application-oriented architecture , 2016, Multimedia Tools and Applications.

[16]  Rob Fergus,et al.  Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks , 2015, NIPS.

[17]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[18]  Xiaogang Wang,et al.  Deep Learning Face Representation from Predicting 10,000 Classes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Jean-Luc Dugelay,et al.  Inpainting of sparse occlusion in face recognition , 2012, 2012 19th IEEE International Conference on Image Processing.

[20]  Wotao Yin,et al.  An Iterative Regularization Method for Total Variation-Based Image Restoration , 2005, Multiscale Model. Simul..

[21]  Vishal M. Patel,et al.  GP-GAN: Gender Preserving GAN for Synthesizing Faces from Landmarks , 2017, 2018 24th International Conference on Pattern Recognition (ICPR).

[22]  Robert Pless,et al.  Deep Feature Interpolation for Image Content Changes , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Kun Zhou,et al.  Face inpainting based on high-level facial attributes , 2017, Comput. Vis. Image Underst..

[24]  Jian Yang,et al.  Learning robust and discriminative low-rank representations for face recognition with occlusion , 2017, Pattern Recognit..

[25]  Yoshua Bengio,et al.  Generative Adversarial Networks , 2014, ArXiv.

[26]  S. Dwivedi,et al.  Obesity May Be Bad: Compressed Convolutional Networks for Biomedical Image Segmentation , 2020 .

[27]  Zhenan Sun,et al.  DeMeshNet: Blind Face Inpainting for Deep MeshFace Verification , 2016, IEEE Transactions on Information Forensics and Security.

[28]  Chi-Keung Tang,et al.  Conditional CycleGAN for Attribute Guided Face Image Generation , 2017, ArXiv.

[29]  Xiaoou Tang,et al.  Surpassing Human-Level Face Verification Performance on LFW with GaussianFace , 2014, AAAI.

[30]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).