Deblurring via Stochastic Refinement

Image deblurring is an ill-posed problem with multiple plausible solutions for a given input image. However, most existing methods produce a deterministic estimate of the clean image and are trained to minimize pixel-level distortion. These metrics are known to be poorly correlated with human perception, and often lead to unrealistic reconstructions. We present an alternative framework for blind deblurring based on conditional diffusion models. Unlike existing techniques, we train a stochastic sampler that refines the output of a deterministic predictor and is capable of producing a diverse set of plausible reconstructions for a given input. This leads to a significant improvement in perceptual quality over existing state-of-the-art methods across multiple standard benchmarks. Our predict-and-refine approach also enables much more efficient sampling compared to typical diffusion models. Combined with a carefully tuned network architecture and inference procedure, our method is competitive in terms of distortion metrics such as PSNR. These results show clear benefits of our diffusion-based method for deblurring and challenge the widely used strategy of producing a single, deterministic reconstruction.

[1]  Jichun Li,et al.  Perceptual Variousness Motion Deblurring with Light Global Context Refinement , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[2]  Alexandros G. Dimakis,et al.  Robust Compressed Sensing MRI with Deep Generative Priors , 2021, NeurIPS.

[3]  Ron Meir,et al.  A Theory of the Distortion-Perception Tradeoff in Wasserstein Space , 2021, NeurIPS.

[4]  Alexandros G. Dimakis,et al.  Fairness for Image Generation with Uncertain Sensitive Attributes , 2021, ICML.

[5]  Heiga Zen,et al.  WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis , 2021, Interspeech.

[6]  Mohammad Norouzi,et al.  Learning to Efficiently Sample from Diffusion Probabilistic Models , 2021, ArXiv.

[7]  Kun Gao,et al.  NTIRE 2021 Learning the Super-Resolution Space Challenge , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[8]  Zhifeng Kong,et al.  On Fast Sampling of Diffusion Probabilistic Models , 2021, ArXiv.

[9]  Michael Elad,et al.  SNIPS: Solving Noisy Inverse Problems Stochastically , 2021, NeurIPS.

[10]  Tal Kachman,et al.  Gotta Go Fast When Generating Data with Score-Based Models , 2021, ArXiv.

[11]  J. Zhang,et al.  HINet: Half Instance Normalization Network for Image Restoration , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[12]  Qi Li,et al.  SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models , 2021, Neurocomputing.

[13]  David J. Fleet,et al.  Image Super-Resolution via Iterative Refinement , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Lior Wolf,et al.  Noise Estimation for Generative Diffusion Models , 2021, ArXiv.

[15]  Michael Elad,et al.  High Perceptual Quality Image Denoising with a Posterior Sampling CGAN , 2021, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[16]  Jimmy S. J. Ren,et al.  Deblurring Dynamic Scenes via Spatially Varying Recurrent Neural Networks , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Ling Shao,et al.  Multi-Stage Progressive Image Restoration , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Michael Elad,et al.  Stochastic Image Denoising by Sampling from the Posterior Distribution , 2021, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[19]  Yen-Yu Lin,et al.  BANet: A Blur-Aware Attention Network for Dynamic Scene Deblurring , 2021, IEEE Transactions on Image Processing.

[20]  P. Milanfar,et al.  Projected Distribution Loss for Image Enhancement , 2020, 2021 IEEE International Conference on Computational Photography (ICCP).

[21]  Peyman Milanfar,et al.  Polyblur: Removing Mild Blur by Polynomial Reblurring , 2020, IEEE Transactions on Computational Imaging.

[22]  Jiaming Song,et al.  Denoising Diffusion Implicit Models , 2020, ICLR.

[23]  Heiga Zen,et al.  WaveGrad: Estimating Gradients for Waveform Generation , 2020, ICLR.

[24]  Sunghyun Cho,et al.  Real-World Blur Dataset for Learning and Benchmarking Deblurring Algorithms , 2020, ECCV.

[25]  L. Gool,et al.  SRFlow: Learning the Super-Resolution Space with Normalizing Flow , 2020, ECCV.

[26]  Pieter Abbeel,et al.  Denoising Diffusion Probabilistic Models , 2020, NeurIPS.

[27]  Eirikur Agustsson,et al.  High-Fidelity Generative Image Compression , 2020, NeurIPS.

[28]  A. Dimakis,et al.  Robust compressed sensing using generative models , 2020, NeurIPS.

[29]  Alexandros G. Dimakis,et al.  Deep Learning Techniques for Inverse Problems in Imaging , 2020, IEEE Journal on Selected Areas in Information Theory.

[30]  A. N. Rajagopalan,et al.  Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  A. Dimakis,et al.  Composing Normalizing Flows for Inverse Problems , 2020, ICML.

[32]  Ling Shao,et al.  Human-Aware Motion Deblurring , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[33]  Zhangyang Wang,et al.  DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[34]  Xiaoyong Shen,et al.  Dynamic Scene Deblurring With Parameter Selective Sharing and Nested Skip Connections , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Guixu Zhang,et al.  Blind Image Deblurring With Local Maximum Gradient Prior , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Hongdong Li,et al.  Deep Stacked Hierarchical Multi-Patch Network for Image Deblurring , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[38]  Stefan Roth,et al.  Normalized Blind Deconvolution , 2018, ECCV.

[39]  Sina Honari,et al.  Distribution Matching Losses Can Hallucinate Features in Medical Image Translation , 2018, MICCAI.

[40]  Lihi Zelnik-Manor,et al.  Maintaining Natural Image Statistics with the Contextual Loss , 2018, ACCV.

[41]  Lihi Zelnik-Manor,et al.  The Contextual Loss for Image Transformation with Non-Aligned Data , 2018, ECCV.

[42]  Ali Ahmed,et al.  Blind Image Deconvolution Using Deep Generative Priors , 2018, IEEE Transactions on Computational Imaging.

[43]  Yi Wang,et al.  Scale-Recurrent Network for Deep Image Deblurring , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[44]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[45]  Arthur Gretton,et al.  Demystifying MMD GANs , 2018, ICLR.

[46]  Jiri Matas,et al.  DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[47]  Y. Blau,et al.  The Perception-Distortion Tradeoff , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[48]  Frank Hutter,et al.  Decoupled Weight Decay Regularization , 2017, ICLR.

[49]  Shubham Pachori,et al.  Deep Generative Filter for Motion Deblurring , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[50]  Bernhard Schölkopf,et al.  Learning Blind Motion Deblurring , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[51]  Eirikur Agustsson,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[52]  Guillermo Sapiro,et al.  Deep Video Deblurring for Hand-Held Cameras , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[54]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[55]  Alexandros G. Dimakis,et al.  Compressed Sensing using Generative Models , 2017, ICML.

[56]  Kyoung Mu Lee,et al.  Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[58]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Narendra Ahuja,et al.  A Comparative Study for Single Image Blind Deblurring , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  Ayan Chakrabarti,et al.  A Neural Approach to Blind Motion Deblurring , 2016, ECCV.

[61]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[62]  Guillermo Sapiro,et al.  Removing Camera Shake via Weighted Fourier Burst Accumulation , 2015, IEEE Transactions on Image Processing.

[63]  Surya Ganguli,et al.  Deep Unsupervised Learning using Nonequilibrium Thermodynamics , 2015, ICML.

[64]  Jean Ponce,et al.  Learning a convolutional neural network for non-uniform motion blur removal , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[65]  Aaron C. Courville,et al.  Generative adversarial networks , 2014, Commun. ACM.

[66]  Li Xu,et al.  Unnatural L0 Sparse Representation for Natural Image Deblurring , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[67]  Alan C. Bovik,et al.  Making a “Completely Blind” Image Quality Analyzer , 2013, IEEE Signal Processing Letters.

[68]  Xiang Zhu,et al.  Deconvolving PSFs for a Better Motion Deblurring Using Multiple Images , 2012, ECCV.

[69]  Frédo Durand,et al.  Understanding and evaluating blind deconvolution algorithms , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[70]  Jiaya Jia,et al.  High-quality motion deblurring from a single image , 2008, ACM Trans. Graph..

[71]  William T. Freeman,et al.  Removing camera shake from a single photograph , 2006, ACM Trans. Graph..

[72]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[73]  Tony F. Chan,et al.  Total variation blind deconvolution , 1998, IEEE Trans. Image Process..

[74]  Florian Jug,et al.  Fully Unsupervised Diversity Denoising with Convolutional Variational Autoencoders , 2021, ICLR.

[75]  Tao Qin,et al.  PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Driven Adaptive Prior , 2021, ArXiv.

[76]  Eero P. Simoncelli,et al.  Stochastic Solutions for Linear Inverse Problems using the Prior Implicit in a Denoiser , 2021, NeurIPS.

[77]  Sundaresh Ram,et al.  Removing Camera Shake from a Single Photograph , 2009 .