Rethinking Deep Image Prior for Denoising

Deep image prior (DIP) serves as a good inductive bias for diverse inverse problems. Among them, denoising is known to be particularly challenging for the DIP due to noise fitting with the requirement of an early stopping. To address the issue, we first analyze the DIP by the notion of effective degrees of freedom (DF) to monitor the optimization progress and propose a principled stopping criterion before fitting to noise without access of a paired ground truth image for Gaussian noise. We also propose the ‘stochastic temporal ensemble (STE)’ method for incorporating techniques to further improve DIP’s performance for denoising. We additionally extend our method to Poisson noise. Our empirical validations show that given a single noisy image, our method denoises the image while preserving rich textual details. Further, our approach outperforms prior arts in LPIPS by large margins with comparable PSNR and SSIM on seven different datasets.

[1]  Jaakko Lehtinen,et al.  Noise2Noise: Learning Image Restoration without Clean Data , 2018, ICML.

[2]  Thierry Blu,et al.  Image Denoising in Mixed Poisson–Gaussian Noise , 2011, IEEE Transactions on Image Processing.

[3]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Eero P. Simoncelli,et al.  Learning to be Bayesian without Supervision , 2006, NIPS.

[5]  Mingqin Chen,et al.  Self2Self With Dropout: Learning Self-Supervised Denoising From Single Image , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Dimitri Van De Ville,et al.  SURE-Based Non-Local Means , 2009, IEEE Signal Processing Letters.

[7]  Lei Zhang,et al.  FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising , 2017, IEEE Transactions on Image Processing.

[8]  Kaiming He,et al.  Data Distillation: Towards Omni-Supervised Learning , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[10]  Xiao-Ping Zhang,et al.  Adaptive denoising based on SURE risk , 1998, IEEE Signal Processing Letters.

[11]  Jean-Michel Morel,et al.  Can a Single Image Denoising Neural Network Handle All Levels of Gaussian Noise? , 2014, IEEE Signal Processing Letters.

[12]  Se Young Chun,et al.  Training and Refining Deep Learning Based Denoisers without Ground Truth Data. , 2018 .

[13]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[14]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[15]  Ryan J. Tibshirani,et al.  Degrees of freedom and model search , 2014, 1402.1920.

[16]  B. Efron.  The Estimation of Prediction Error , 2004 .

[17]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[18]  C. Stein.  Estimation of the Mean of a Multivariate Normal Distribution , 1981 .

[19]  I. Johnstone,et al.  Adapting to Unknown Smoothness via Wavelet Shrinkage , 1995 .

[20]  Stefan Harmeling,et al.  Image denoising: Can plain neural networks compete with BM3D? , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Yee Whye Teh,et al.  Bayesian Learning via Stochastic Gradient Langevin Dynamics , 2011, ICML.

[22]  Dimitri Van De Ville,et al.  Nonlocal Means With Dimensionality Reduction and SURE-Based Parameter Selection , 2011, IEEE Transactions on Image Processing.

[23]  Se Young Chun,et al.  Unsupervised Training of Denoisers for Low-Dose CT Reconstruction Without Full-Dose Ground Truth , 2020, IEEE Journal of Selected Topics in Signal Processing.

[24]  Stefan Roth,et al.  Benchmarking Denoising Algorithms with Real Photographs , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Xiangchu Feng,et al.  FOCNet: A Fractional Optimal Control Network for Image Denoising , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Se Young Chun,et al.  Extending Stein's unbiased risk estimator to train deep denoisers with correlated pairs of noisy images , 2019, NeurIPS.

[27]  Yochai Blau,et al.  The Perception-Distortion Tradeoff , 2017, CVPR.

[28]  Ulrich Anders,et al.  Model selection in neural networks , 1999, Neural Networks.

[29]  Reinhard Heckel,et al.  Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks , 2018, ICLR.

[30]  Elsa D. Angelini,et al.  An Unbiased Risk Estimator for Image Denoising in the Presence of Mixed Poisson–Gaussian Noise , 2014, IEEE Transactions on Image Processing.

[31]  Yoshua Bengio,et al.  What regularized auto-encoders learn from the data-generating distribution , 2012, J. Mach. Learn. Res..

[32]  Radu Timofte,et al.  A Brief Review of Image Denoising Algorithms and Beyond , 2019, Inpainting and Denoising Challenges.

[33]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[34]  Yonina C. Eldar,et al.  On Divergence Approximations for Unsupervised Training of Deep Denoisers Based on Stein’s Unbiased Risk Estimator , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[35]  Stefan Roth,et al.  Neural Nearest Neighbors Networks , 2018, NeurIPS.

[36]  David L. Donoho,et al.  De-noising by soft-thresholding , 1995, IEEE Trans. Inf. Theory.

[37]  Yonina C. Eldar.  Generalized SURE for Exponential Families: Applications to Regularization , 2008, IEEE Transactions on Signal Processing.

[38]  Pascal Vincent,et al.  Contractive Auto-Encoders: Explicit Invariance During Feature Extraction , 2011, ICML.

[39]  Tianxiang Gao,et al.  Degrees of Freedom in Deep Neural Networks , 2016, UAI.

[40]  Richard G. Baraniuk,et al.  Unsupervised Learning with Stein's Unbiased Risk Estimator , 2018, ArXiv.

[41]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[42]  Liyuan Liu,et al.  On the Variance of the Adaptive Learning Rate and Beyond , 2019, ICLR.

[43]  Alessandro Foi,et al.  Optimal Inversion of the Generalized Anscombe Transformation for Poisson-Gaussian Noise , 2013, IEEE Transactions on Image Processing.

[44]  Nick Moran,et al.  Noisier2Noise: Learning to Denoise From Unpaired Noisy Data , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  OsherStanley,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[46]  Subhransu Maji,et al.  A Bayesian Perspective on the Deep Image Prior , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Wangmeng Zuo,et al.  Learning Deep CNN Denoiser Prior for Image Restoration , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[49]  R. Tibshirani,et al.  Generalized Additive Models , 1986 .

[50]  Thierry Blu,et al.  Monte-Carlo Sure: A Black-Box Optimization of Regularization Parameters for General Denoising Algorithms , 2008, IEEE Transactions on Image Processing.

[51]  Jaakko Lehtinen,et al.  High-Quality Self-Supervised Deep Image Denoising , 2019, NeurIPS.

[52]  Se Young Chun,et al.  Training Deep Learning based Denoisers without Ground Truth Data , 2018, NeurIPS.

[53]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[55]  Michael Elad,et al.  DeepRED: Deep Image Prior Powered by RED , 2019, ICCV 2019.

[56]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .