Multi-task ConvNet for blind face inpainting with application to face verification

Face verification between ID photos and life photos (FVBIL) is gaining traction with the rapid development of the Internet. However, ID photos provided by the Chinese administration center are often corrupted with wavy lines to prevent misuse, which poses great difficulty to accurate FVBIL. Therefore, this paper tries to improve the verification performance by studying a new problem, i.e. blind face inpainting, where we aim at restoring clean face images from the corrupted ID photos. The term blind indicates that the locations of corruptions are not known in advance. We formulate blind face inpainting as a joint detection and reconstruction problem. A multi-task ConvNet is accordingly developed to facilitate end to end network training for accurate and fast inpainting. The ConvNet is used to (i) regress the residual values between the clean/corrupted ID photo pairs and (ii) predict the positions of residual regions. Moreover, to achieve better inpainting results, we employ a skip connection to fuse information in the intermediate layer. To enable training of our ConvNet, we collect a dataset of synthetic clean/corrupted ID photo pairs with 500 thousand samples from around 10 thousand individuals. Experiments demonstrate that our multi-task ConvNet achieves superior performance in terms of reconstruction errors, convergence speed and verification accuracy.

[1]  Li Xu,et al.  Shepard Convolutional Neural Networks , 2015, NIPS.

[2]  Gang Wang,et al.  Multi-Task CNN Model for Attribute Prediction , 2015, IEEE Transactions on Multimedia.

[3]  Qi Yin,et al.  Naive-Deep Face Recognition: Touching the Limit of LFW Benchmark or Not? , 2015, ArXiv.

[4]  Yueting Zhuang,et al.  DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection , 2015, IEEE Transactions on Image Processing.

[5]  Ran He,et al.  Two-Stage Nonnegative Sparse Representation for Large-Scale Face Recognition , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[6]  Shengcai Liao,et al.  Learning Face Representation from Scratch , 2014, ArXiv.

[7]  Enhong Chen,et al.  Image Denoising and Inpainting with Deep Neural Networks , 2012, NIPS.

[8]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[9]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[10]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Shiliang Sun,et al.  Multitask Twin Support Vector Machines , 2012, ICONIP.

[13]  Andrew Zisserman,et al.  Speeding up Convolutional Neural Networks with Low Rank Expansions , 2014, BMVC.

[14]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[15]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[16]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Rob Fergus,et al.  Restoring an Image Taken through a Window Covered with Dirt or Rain , 2013, 2013 IEEE International Conference on Computer Vision.

[19]  Xiaoou Tang,et al.  Facial Landmark Detection by Deep Multi-task Learning , 2014, ECCV.

[20]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[21]  Zhenan Sun,et al.  A Lightened CNN for Deep Face Representation , 2015, ArXiv.

[22]  Stefan Harmeling,et al.  Image denoising: Can plain neural networks compete with BM3D? , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Tieniu Tan,et al.  Learning Symmetry Features for Face Detection Based on Sparse Group Lasso , 2013, CCBR.