Facial Image Synthesis and Super-Resolution With Stacked Generative Adversarial Network

Abstract Image synthesis and super-resolution (SR) have always been a hot spot for computer vision and image processing research. Since the development of Deep Learning, especially after the Deep Convolutional Generative Adversarial Network (DC-GAN) methods, facial image synthesis and SR problem had been solved in many circumstances. But most of the existing works were focused on natural-looking of the synthesized result rather than keeping facial information of the original image. Our paper presented an end-to-end method of getting high-resolution photo-realistic facial images from low-resolution (LR) in-the-wild images without losing the facial identity details. The pipeline used a flexible stacked GAN structure for the SR process with different target image resolutions on different upscaling factors. To avoid getting blur or nonsensical image output and realize the flexibility, “U-Net” architecture and upsampling layers with residual learning blocks were stacked. The stacked network structure makes applying different loss functions in different parts of the network possible, which helps to solve the two problems of keeping identical facial details of the LR input image and generating high-quality output images simultaneously. By using 3 different loss functions in different positions of the stacked network separately, through experimental comparison, we found the best stacked residual block parameters which could get the best output image quality. Experimental results also explicated that the network had a good SR ability compare to state of the art methods in different resolution and upscaling factor.

[1]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[3]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[4]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[5]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[6]  Qingxiang Wu,et al.  Image super-resolution using a dilated convolutional neural network , 2018, Neurocomputing.

[7]  Yi Wang,et al.  Scale-Recurrent Network for Deep Image Deblurring , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Jing Yang,et al.  To learn image super-resolution, use a GAN to learn how to do image degradation first , 2018, ECCV.

[9]  Dwarikanath Mahapatra,et al.  Image super-resolution using progressive generative adversarial networks for medical image analysis , 2019, Comput. Medical Imaging Graph..

[10]  Xuelong Li,et al.  Stacked Fisher autoencoder for SAR change detection , 2019, Pattern Recognit..

[11]  Ponnuthurai N. Suganthan,et al.  High-quality face image generated with conditional boundary equilibrium generative adversarial networks , 2018, Pattern Recognit. Lett..

[12]  Xiaogang Wang,et al.  StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[14]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[17]  Jung-Woo Ha,et al.  StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Sridha Sridharan,et al.  Super-resolution for biometrics: A comprehensive survey , 2018, Pattern Recognit..

[19]  Meng Yang,et al.  Triple-translation GAN with multi-layer sparse representation for face image synthesis , 2019, Neurocomputing.

[20]  Yang Chen,et al.  Recovering Super-Resolution Generative Adversarial Network for Underwater Images , 2019, ICONIP.

[21]  Tieniu Tan,et al.  Wavelet-SRNet: A Wavelet-Based CNN for Multi-scale Face Super Resolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[22]  Joan Bruna,et al.  Super-Resolution with Deep Convolutional Sufficient Statistics , 2015, ICLR.

[23]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Xinfeng Zhang,et al.  Multiple Cycle-in-Cycle Generative Adversarial Networks for Unsupervised Image Super-Resolution , 2020, IEEE Transactions on Image Processing.

[25]  Mario Lucic,et al.  Are GANs Created Equal? A Large-Scale Study , 2017, NeurIPS.

[26]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[27]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[28]  Jian Yang,et al.  FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Yi Sun,et al.  FG-SRGAN: A Feature-Guided Super-Resolution Generative Adversarial Network for Unpaired Image Super-Resolution , 2019, ISNN.

[30]  Xiaoou Tang,et al.  Deep Cascaded Bi-Network for Face Hallucination , 2016, ECCV.

[31]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).