Integrating Neural Networks Into the Blind Deblurring Framework to Compete With the End-to-End Learning-Based Methods

Recently, the end-to-end learning-based methods have been proven effective for the blind image deblurring. Without human-made assumptions or numerical algorithms, they are able to restore images with fewer artifacts and better perceptual quality. However, in practice, these methods suffer from limited performance under complex motion scenario and produces unnatural results sometimes. In this paper, in order to overcome their limitations, we propose to integrate deep convolution neural networks into a conventional deblurring framework. Specifically, we propose Stacked Estimation Residual Net (SEN) to estimate the motion flow map and Recurrent Prior Generative and Adversarial Net (RP-GAN) to learn the implicit image prior for the optimization. Comparing with the state-of-the-art end-to-end learning-based methods, the proposed method restores image content more naturally and shows better generalization ability.

[1]  Frédo Durand,et al.  Understanding and evaluating blind deconvolution algorithms , 2009, CVPR.

[2]  Stefan Harmeling,et al.  Image denoising: Can plain neural networks compete with BM3D? , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Ian D. Reid,et al.  From Motion Blur to Motion Flow: A Deep Learning Solution for Removing Heterogeneous Motion Blur , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Daniele Perrone,et al.  Total Variation Blind Deconvolution: The Devil Is in the Details , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Renjie Liao,et al.  Detail-Revealing Deep Video Super-Resolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[6]  Yi Wang,et al.  Scale-Recurrent Network for Deep Image Deblurring , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[7]  Sunghyun Cho,et al.  Edge-based blur kernel estimation using patch priors , 2013, IEEE International Conference on Computational Photography (ICCP).

[8]  Matthias Zwicker,et al.  Image Restoration using Autoencoding Priors , 2017, VISIGRAPP.

[9]  Jiri Matas,et al.  DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[10]  Yanning Zhang,et al.  Multi-image Blind Deblurring Using a Coupled Adaptive Sparse Prior , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Li Xu,et al.  Unnatural L0 Sparse Representation for Natural Image Deblurring , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[13]  Rob Fergus,et al.  Blind deconvolution using a normalized sparsity measure , 2011, CVPR 2011.

[14]  Mingkui Tan,et al.  Blind Image Deconvolution by Automatic Gradient Activation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Bernhard Schölkopf,et al.  Learning to Deblur , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Frédo Durand,et al.  Efficient marginal likelihood optimization in blind deconvolution , 2011, CVPR 2011.

[18]  Wangmeng Zuo,et al.  Learning Deep CNN Denoiser Prior for Image Restoration , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Sunghyun Cho,et al.  Fast motion deblurring , 2009, SIGGRAPH 2009.

[20]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[21]  Thekke Madam Nimisha,et al.  Blur-Invariant Deep Learning for Blind-Deblurring , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[22]  Guangming Shi,et al.  Denoising Prior Driven Deep Neural Network for Image Restoration , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Jean Ponce,et al.  Learning a convolutional neural network for non-uniform motion blur removal , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Tony F. Chan,et al.  Total variation blind deconvolution , 1998, IEEE Trans. Image Process..

[25]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[26]  Marcin Andrychowicz,et al.  Hindsight Experience Replay , 2017, NIPS.

[27]  Bernhard Schölkopf,et al.  A Machine Learning Approach for Non-blind Image Deconvolution , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Ming-Hsuan Yang,et al.  Learning a Discriminative Prior for Blind Image Deblurring , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Jia Deng,et al.  Stacked Hourglass Networks for Human Pose Estimation , 2016, ECCV.

[30]  Ling Shao,et al.  Blind Image Blur Estimation via Deep Learning , 2016, IEEE Transactions on Image Processing.

[31]  Deqing Sun,et al.  Blind Image Deblurring Using Dark Channel Prior , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Bernhard Schölkopf,et al.  Recording and Playback of Camera Shake: Benchmarking Blind Deconvolution with a Real-World Database , 2012, ECCV.

[33]  Ning Xu,et al.  Deep Image Matting , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.

[35]  Rynson W. H. Lau,et al.  Learning Fully Convolutional Networks for Iterative Non-blind Deconvolution , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Tae Hyun Kim,et al.  Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Sundaresh Ram,et al.  Removing Camera Shake from a Single Photograph , 2009 .

[38]  Charles A. Bouman,et al.  Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolation , 2015, IEEE Transactions on Computational Imaging.

[39]  Xiaoou Tang,et al.  Video Frame Synthesis Using Deep Voxel Flow , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[40]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[41]  Guillermo Sapiro,et al.  Deep Video Deblurring , 2016, ArXiv.