CG-GAN: Class-Attribute Guided Generative Adversarial Network for Old Photo Restoration

Old photos are an important carrier to preserve the past. Usually, the degradation of old photos is rather diverse and complex. Therefore, the existing methods to solve conventional restoration tasks are difficult to generalize. To solve this problem, we propose a novel method based on generative adversarial network. Our method utilizes the class-attributes of old photos to complete restoration in latent space. Specifically, we divide the process of restoring old photos into two stages, one is global defect restoration stage and the other is local detail restoration stage. In global defect restoration stage, we extract the latent representations of four classes of high-level attributes that are smoothness, clarity, connectivity and completeness. We use latent class-attribute information to restore global defects in latent space and we obtain conditional control vector through a condition network to guide the subsequent local detail restoration stage. In local detail restoration stage, we propose a dynamic condition-guided restoration module that selects the most suitable combination of features to further restore local details through a dynamic network. In addition, we propose a dual discriminator to pay more attention to style and defect restoration. We ignore the complex degradation of old photos to directly restore advanced class-attributes. Therefore, our method has better generalization performance. Experiments show that our method is superior to other existing methods of image restoration in terms of visual quality and numerical metrics.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Aude Oliva,et al.  GANalyze: Toward Visual Definitions of Cognitive Image Properties , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[3]  Wangmeng Zuo,et al.  Learning Deep CNN Denoiser Prior for Image Restoration , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[5]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[6]  Zhan Xu,et al.  Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Masanori Suganuma,et al.  Attention-Based Adaptive Selection of Operations for Image Restoration in the Presence of Unknown Combined Distortions , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Radu Timofte,et al.  Unsupervised Learning for Real-World Super-Resolution , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[10]  Feiyue Huang,et al.  Real-World Super-Resolution via Kernel Estimation and Noise Injection , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[11]  Phillip Isola,et al.  On the "steerability" of generative adversarial networks , 2019, ICLR.

[12]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[13]  Jungwon Lee,et al.  DN-ResNet: Efficient Deep Residual Network for Image Denoising , 2018, ACCV.

[14]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[15]  Xin Wang,et al.  SkipNet: Learning Dynamic Routing in Convolutional Networks , 2017, ECCV.

[16]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[17]  William T. Freeman,et al.  What makes a good model of natural images? , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Quoc V. Le,et al.  CondConv: Conditionally Parameterized Convolutions for Efficient Inference , 2019, NeurIPS.

[19]  Yi Wang,et al.  VCNet: A Robust Approach to Blind Image Inpainting , 2020, ECCV.

[20]  Bo Zhang,et al.  Bringing Old Photos Back to Life , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[22]  Jung-Woo Ha,et al.  StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[23]  Stan Z. Li Markov Random Field Modeling in Image Analysis , 2009, Advances in Pattern Recognition.

[24]  Yui-Lam Chan,et al.  Unsupervised Real Image Super-Resolution via Generative Variational AutoEncoder , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[25]  Aggelos K. Katsaggelos,et al.  Total variation super resolution using a variational approach , 2008, 2008 15th IEEE International Conference on Image Processing.

[26]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Deli Zhao,et al.  In-Domain GAN Inversion for Real Image Editing , 2020, ECCV.

[28]  Ting-Chun Wang,et al.  Image Inpainting for Irregular Holes Using Partial Convolutions , 2018, ECCV.

[29]  Yoshua Bengio,et al.  Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation , 2013, ArXiv.

[30]  Prabir Kumar Biswas,et al.  Prior Guided GAN Based Semantic Inpainting , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Liang Lin,et al.  Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[32]  Michal Irani,et al.  Blind Super-Resolution Kernel Estimation using an Internal-GAN , 2019, NeurIPS.

[33]  Zibo Meng,et al.  Residual Channel Attention Generative Adversarial Network for Image Super-Resolution and Noise Reduction , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[34]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[35]  Guillermo Sapiro,et al.  Non-local sparse models for image restoration , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[36]  Victor M. Becerra,et al.  Dynamic Neural Networks , 2003 .

[37]  Radu Timofte,et al.  Frequency Separation for Real-World Super-Resolution , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[38]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[39]  Bolei Zhou,et al.  Interpreting the Latent Space of GANs for Semantic Face Editing , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[41]  Bolei Zhou,et al.  Image Processing Using Multi-Code GAN Prior , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Misha Denil,et al.  Predicting Parameters in Deep Learning , 2014 .

[43]  Mehran Ebrahimi,et al.  EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning , 2019, ArXiv.