Reference Based Face Super-Resolution

Despite the great progress of image super-resolution in recent years, face super-resolution has still much room to explore good visual quality while preserving original facial attributes for larger up-scaling factors. This paper investigates a new research direction in face super-resolution, called Reference based face Super-Resolution (RefSR), in which a reference facial image containing genuine attributes is provided in addition to the low-resolution images for super-resolution. We focus on transferring the key information extracted from reference facial images to the super-resolution process to guarantee the content similarity between the reference and super-resolution image. We propose a novel Conditional Variational AutoEncoder model for this Reference based Face Super-Resolution (RefSR-VAE). By using the encoder to map the reference image to the joint latent space, we can then use the decoder to sample the encoder results to super-resolve low-resolution facial images to generate super-resolution images with good visual quality. We create a benchmark dataset on reference based face super-resolution (RefSR-Face) for general research use, which contains reference images paired with low-resolution images of various pose, emotions, ages and appearance. Both objective and subjective evaluations were conducted, which demonstrate the great potential of using reference images for face super-resolution. By comparing it with state-of-the-art super-resolution approaches, our proposed approach also achieves superior performance.

[1]  Leon A. Gatys,et al.  A Neural Algorithm of Artistic Style , 2015, ArXiv.

[2]  Guangming Shi,et al.  Nonlocal Sparse and Low-Rank Regularization for Optical Flow Estimation , 2014, IEEE Transactions on Image Processing.

[3]  David Pfau,et al.  Unrolled Generative Adversarial Networks , 2016, ICLR.

[4]  Gang Hua,et al.  Labeled Faces in the Wild: A Survey , 2016 .

[5]  Reuben A. Farrugia,et al.  Face Hallucination Using Linear Models of Coupled Sparse Support , 2015, IEEE Transactions on Image Processing.

[6]  Xin Yu,et al.  Hallucinating Very Low-Resolution Unaligned and Noisy Face Images by Transformative Discriminative Autoencoders , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[8]  Yui-Lam Chan,et al.  Fast image super-resolution via Randomized Multi-split Forests , 2017, 2017 IEEE International Symposium on Circuits and Systems (ISCAS).

[9]  Wan-Chi Siu,et al.  Hierarchical Back Projection Network for Image Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[10]  Ming-Hsuan Yang,et al.  Generative Face Completion , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Thomas S. Huang,et al.  Interactive Facial Feature Localization , 2012, ECCV.

[12]  Xin Yu,et al.  Ultra-Resolving Face Images by Discriminative Generative Networks , 2016, ECCV.

[13]  Xin Yu,et al.  Face Super-Resolution Guided by Facial Component Heatmaps , 2018, ECCV.

[14]  Rama Chellappa,et al.  Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models , 2018, ICLR.

[15]  Wei Liu,et al.  Super-Identity Convolutional Neural Network for Face Hallucination , 2018, ECCV.

[16]  Omkar M. Parkhi,et al.  VGGFace2: A Dataset for Recognising Faces across Pose and Age , 2017, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[17]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[18]  Alessandro Foi,et al.  Spatially adaptive alpha-rooting in BM3D sharpening , 2011, Electronic Imaging.

[19]  Gregory Shakhnarovich,et al.  Deep Back-Projection Networks for Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  Seyed-Mohsen Moosavi-Dezfooli,et al.  DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[22]  Wan-Chi Siu,et al.  Image super-resolution via hybrid NEDI and wavelet-based scheme , 2015, 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA).

[23]  Yui-Lam Chan,et al.  Joint Back Projection and Residual Networks for Efficient Image Super-Resolution , 2018, 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC).

[24]  Ruimin Hu,et al.  Face Super-Resolution via Multilayer Locality-Constrained Iterative Neighbor Embedding and Intermediate Dictionary Learning , 2014, IEEE Transactions on Image Processing.

[25]  Wan-Chi Siu,et al.  Image super-resolution via weighted random forest , 2017, 2017 IEEE International Conference on Industrial Technology (ICIT).

[26]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  John R. Hershey,et al.  Global-Local Face Upsampling Network , 2016, ArXiv.

[28]  M. E. Celebi,et al.  Advances in Face Detection and Facial Image Analysis , 2016 .

[29]  Hong Chang,et al.  Super-resolution through neighbor embedding , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[30]  Wan-Chi Siu,et al.  Learning Hierarchical Decision Trees for Single-Image Super-Resolution , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[31]  Ruigang Yang,et al.  Learning Warped Guidance for Blind Face Restoration , 2018, ECCV.

[32]  Kyoung Mu Lee,et al.  Deeply-Recursive Convolutional Network for Image Super-Resolution , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[34]  Jian Yang,et al.  FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  Yochai Blau,et al.  The Perception-Distortion Tradeoff , 2017, CVPR.

[36]  Wan-Chi Siu,et al.  Cascaded Random Forests for Fast Image Super-Resolution , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[37]  Luc Van Gool,et al.  Anchored Neighborhood Regression for Fast Example-Based Super-Resolution , 2013, 2013 IEEE International Conference on Computer Vision.

[38]  Ruimin Hu,et al.  Noise Robust Face Hallucination via Locality-Constrained Representation , 2014, IEEE Transactions on Multimedia.

[39]  Xiaoou Tang,et al.  Deep Cascaded Bi-Network for Face Hallucination , 2016, ECCV.

[40]  Adam Roberts,et al.  Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models , 2017, ICLR.

[41]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[42]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[43]  Guoying Zhao,et al.  Hallucinating Face Image by Regularization Models in High-Resolution Feature Space , 2018, IEEE Transactions on Image Processing.

[44]  Hairong Qi,et al.  Image Super-Resolution by Neural Texture Transfer , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[46]  Luc Van Gool,et al.  A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution , 2014, ACCV.

[47]  Tania Stathaki,et al.  SRHRF+: Self-Example Enhanced Single Image Super-Resolution Using Hierarchical Random Forests , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[48]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Ariel D. Procaccia,et al.  Variational Dropout and the Local Reparameterization Trick , 2015, NIPS.

[50]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).