Recovering Extremely Degraded Faces by Joint Super-Resolution and Facial Composite

In the past a few years, we witnessed rapid advancement in face super-resolution from very low resolution(VLR) images. However, most of the previous studies focus on solving such problem without explicitly considering the impact of severe real-life image degradation (e.g. blur and noise). We can show that robustly recover details from VLR images is a task beyond the ability of current state-of-the-art method. In this paper, we borrow ideas from "facial composite" and propose an alternative approach to tackle this problem. We endow the degraded VLR images with additional cues by integrating existing face components from multiple reference images into a novel learning pipeline with both low level and high level semantic loss function as well as a specialized adversarial based training scheme. We show that our method is able to effectively and robustly restore relevant facial details from 16x16 images with extreme degradation. We also tested our approach against real-life images and our method performs favorably against previous methods.

[1]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Jian Yang,et al.  FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3]  Kin-Man Lam,et al.  A novel face-hallucination scheme based on singular value decomposition , 2013, Pattern Recognit..

[4]  Ming-Hsuan Yang,et al.  Generative Face Completion , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Xin Yu,et al.  Face Super-Resolution Guided by Facial Component Heatmaps , 2018, ECCV.

[6]  Xiaoou Tang,et al.  Deep Cascaded Bi-Network for Face Hallucination , 2016, ECCV.

[7]  Liang Lin,et al.  Attention-Aware Face Hallucination via Deep Reinforcement Learning , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[9]  Georgios Tzimiropoulos,et al.  How Far are We from Solving the 2D & 3D Face Alignment Problem? (and a Dataset of 230,000 3D Facial Landmarks) , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[10]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Luc Van Gool,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[12]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[13]  Ruigang Yang,et al.  Learning Warped Guidance for Blind Face Restoration , 2018, ECCV.

[14]  Ming-Hsuan Yang,et al.  Joint Face Hallucination and Deblurring via Structure Generation and Detail Enhancement , 2018, International Journal of Computer Vision.

[15]  Xin Yu,et al.  Hallucinating Very Low-Resolution Unaligned and Noisy Face Images by Transformative Discriminative Autoencoders , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[17]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[18]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Jing Yang,et al.  To learn image super-resolution, use a GAN to learn how to do image degradation first , 2018, ECCV.

[20]  Yilong Yin,et al.  Multi-view face hallucination using SVD and a mapping model , 2019, Inf. Sci..

[21]  Thomas S. Huang,et al.  Image Super-Resolution via Dual-State Recurrent Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[23]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[24]  Georgios Tzimiropoulos,et al.  Super-FAN: Integrated Facial Landmark Localization and Super-Resolution of Real-World Low Resolution Faces in Arbitrary Poses with GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Xin Yu,et al.  Super-Resolving Very Low-Resolution Face Images with Supplementary Attributes , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Tieniu Tan,et al.  Wavelet-SRNet: A Wavelet-Based CNN for Multi-scale Face Super Resolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[27]  Tim Valentine,et al.  Facial Composites: Forensic Utility and Psychological Research , 2007 .

[28]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[29]  Kin-Man Lam,et al.  Simultaneous Hallucination and Recognition of Low-Resolution Faces Based on Singular Value Decomposition , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[30]  Scott Schaefer,et al.  Image deformation using moving least squares , 2006, ACM Trans. Graph..

[31]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[32]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[33]  Jiawei Zhang,et al.  Learning to Hallucinate Face Images via Component Generation and Enhancement , 2017, IJCAI.