An Image Restoration Architecture using Abstract Features and Generative Models

Image restoration and its different variations are important topics in low-level image processing. One of the main challenges in image restoration is dependency of current methods to the corruption characteristics. In this paper, we have proposed an image restoration architecture that enables us to address different types of corruption, regardless of type, amount and location. The main intuition behind our approach is restoring original images from abstracted perceptual features. Using an encoder-decoder architecture, image restoration can be defined as an image transformation task. Abstraction of perceptual features is done in the encoder part of the model and determines the sampling point within original images' Probability Density Function (PDF). The PDF of original images is learned in the decoder section by using a Generative Adversarial Network (GAN) that receives the sampling point from the encoder part. Concretely, sampling from the learned PDF restores original image from its corrupted version. Pretrained network extracts perceptual features and Restricted Boltzmann Machine (RBM) makes the abstraction over them in the encoder section. By developing a new algorithm for training the RBM, the features of the corrupted images have been refined. In the decoder, the Generator network restores original images from abstracted perceptual features while Discriminator determines how good the restoration result is. The proposed approach has been compared with both traditional approaches like BM3D and with modern deep models like IRCNN and NCSR. We have also considered three different categories of corruption including denoising, inpainting and deblurring. Experimental results confirm performance of the model.

[1]  G. Moustakides,et al.  Image Restoration from Parametric Transformations using Generative Models , 2020, ArXiv.

[2]  Léon Bottou,et al.  Towards Principled Methods for Training Generative Adversarial Networks , 2017, ICLR.

[3]  José M. Bioucas-Dias,et al.  Adaptive total variation image deblurring: A majorization-minimization approach , 2009, Signal Process..

[4]  Lina J. Karam,et al.  A Study and Comparison of Human and Deep Learning Recognition Performance under Visual Distortions , 2017, 2017 26th International Conference on Computer Communication and Networks (ICCCN).

[5]  Yasemin Altun,et al.  Machine Learning Methods for Automatic Image Colorization , 2011 .

[6]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[7]  Hao Li,et al.  High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[10]  Zhijing Yang,et al.  Blind inpainting using the fully convolutional neural network , 2017, The Visual Computer.

[11]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[12]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[13]  H. Sebastian Seung,et al.  Natural Image Denoising with Convolutional Networks , 2008, NIPS.

[14]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[15]  Hassan Farsi,et al.  Image Restoration by Variable Splitting based on Total Variant Regularizer , 2018 .

[16]  Mohammed Atiquzzaman,et al.  Automated Colorization of a Grayscale Image With Seed Points Propagation , 2020, IEEE Transactions on Multimedia.

[17]  Karen O. Egiazarian,et al.  Color Image Denoising via Sparse 3D Collaborative Filtering with Grouping Constraint in Luminance-Chrominance Space , 2007, 2007 IEEE International Conference on Image Processing.

[18]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[19]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[20]  João Paulo Papa,et al.  A Robust Restricted Boltzmann Machine for Binary Image Denoising , 2017, 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI).

[21]  Enhong Chen,et al.  Image Denoising and Inpainting with Deep Neural Networks , 2012, NIPS.

[22]  Jefersson Alex dos Santos,et al.  Towards better exploiting convolutional neural networks for remote sensing scene classification , 2016, Pattern Recognit..

[23]  Yu-Bin Yang,et al.  Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections , 2016, ArXiv.

[24]  Xiaoou Tang,et al.  Compression Artifacts Reduction by a Deep Convolutional Network , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[25]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[27]  Matthias Bethge,et al.  A note on the evaluation of generative models , 2015, ICLR.

[28]  Raveendran Paramesran,et al.  Hybrid non-convex second-order total variation with applications to non-blind image deblurring , 2020, Signal Image Video Process..

[29]  Yun Fu,et al.  Residual Dense Network for Image Restoration , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Chen Change Loy,et al.  Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation , 2020, ECCV.

[31]  Wangmeng Zuo,et al.  Learning Deep CNN Denoiser Prior for Image Restoration , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[33]  Xianming Liu,et al.  Connecting Image Denoising and High-Level Vision Tasks via Deep Learning , 2018, IEEE Transactions on Image Processing.

[34]  Jie Li,et al.  WaterGAN: Unsupervised Generative Network to Enable Real-Time Color Correction of Monocular Underwater Images , 2017, IEEE Robotics and Automation Letters.

[35]  Steven C. H. Hoi,et al.  Deep Learning for Image Super-Resolution: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Kristen Grauman,et al.  On-demand Learning for Deep Image Restoration , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[37]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[38]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[39]  Stefan Harmeling,et al.  Image denoising: Can plain neural networks compete with BM3D? , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Lei Zhang,et al.  Image Deblurring and Super-Resolution by Adaptive Sparse Domain Selection and Adaptive Regularization , 2010, IEEE Transactions on Image Processing.

[41]  Geoffrey E. Hinton,et al.  Robust Boltzmann Machines for recognition and denoising , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Uday B. Desai,et al.  Image restoration using a multilayer perceptron with a multilevel sigmoidal function , 1993, IEEE Trans. Signal Process..

[43]  Frédo Durand,et al.  Understanding and evaluating blind deconvolution algorithms , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Chunwei Tian,et al.  Image denoising using deep CNN with batch renormalization , 2020, Neural Networks.

[45]  Lei Zhang,et al.  Nonlocally Centralized Sparse Representation for Image Restoration , 2013, IEEE Transactions on Image Processing.

[46]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Zhijun Zhang,et al.  Weighted Low-Rank Tensor Recovery for Hyperspectral Image Restoration , 2017, IEEE Transactions on Cybernetics.

[48]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[49]  Geoffrey E. Hinton.  A Practical Guide to Training Restricted Boltzmann Machines , 2012, Neural Networks: Tricks of the Trade.

[50]  Andy M. Yip,et al.  Recent Developments in Total Variation Image Restoration , 2004 .