Score-Based Diffusion Models as Principled Priors for Inverse Imaging

Priors are essential for reconstructing images from noisy and/or incomplete measurements. The choice of the prior determines both the quality and uncertainty of recovered images. We propose turning score-based diffusion models into principled image priors ("score-based priors") for analyzing a posterior of images given measurements. Previously, probabilistic priors were limited to handcrafted regularizers and simple distributions. In this work, we empirically validate the theoretically-proven probability function of a score-based diffusion model. We show how to sample from resulting posteriors by using this probability function for variational inference. Our results, including experiments on denoising, deblurring, and interferometric imaging, suggest that score-based priors enable principled inference with a sophisticated, data-driven image prior.

[1]  P. Milanfar,et al.  Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration , 2023, ArXiv.

[2]  Y. Bengio,et al.  Posterior samples of source galaxies in strong gravitational lenses with score-based priors , 2022, ArXiv.

[3]  Michael T. McCann,et al.  Diffusion Posterior Sampling for General Noisy Inverse Problems , 2022, ICLR.

[4]  Michael Elad,et al.  Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance , 2022, Trans. Mach. Learn. Res..

[5]  Jonathan Ho Classifier-Free Diffusion Guidance , 2022, ArXiv.

[6]  D. Samaras,et al.  Diffusion models as plug-and-play priors , 2022, NeurIPS.

[7]  Cheng Lu,et al.  Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matching , 2022, ICML.

[8]  J. C. Ye,et al.  Improving Diffusion Models for Inverse Problems using Manifold Constraints , 2022, Neural Information Processing Systems.

[9]  David J. Fleet,et al.  Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding , 2022, NeurIPS.

[10]  Prafulla Dhariwal,et al.  Hierarchical Text-Conditional Image Generation with CLIP Latents , 2022, ArXiv.

[11]  Patrick Kidger On Neural Differential Equations , 2022, ArXiv.

[12]  Michael Elad,et al.  Denoising Diffusion Restoration Models , 2022, NeurIPS.

[13]  Ming-Hsuan Yang,et al.  Deep Image Deblurring: A Survey , 2022, International Journal of Computer Vision.

[14]  B. Ommer,et al.  High-Resolution Image Synthesis with Latent Diffusion Models , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Prafulla Dhariwal,et al.  GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models , 2021, ICML.

[16]  Jong-Chul Ye,et al.  Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  S. Ermon,et al.  Solving Inverse Problems in Medical Imaging with Score-Based Generative Models , 2021, ICLR.

[18]  Jong-Chul Ye,et al.  Score-based diffusion models for accelerated MRI , 2021, Medical Image Anal..

[19]  Youngjune Gwon,et al.  ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Alexandros G. Dimakis,et al.  Robust Compressed Sensing MRI with Deep Generative Priors , 2021, NeurIPS.

[21]  Diederik P. Kingma,et al.  Variational Diffusion Models , 2021, ArXiv.

[22]  Michael Elad,et al.  SNIPS: Solving Noisy Inverse Problems Stochastically , 2021, NeurIPS.

[23]  David J. Fleet,et al.  Cascaded Diffusion Models for High Fidelity Image Generation , 2021, J. Mach. Learn. Res..

[24]  Adrian V. Dalca,et al.  End-to-End Sequential Sampling and Reconstruction for MRI , 2021, ML4H@NeurIPS.

[25]  Prafulla Dhariwal,et al.  Diffusion Models Beat GANs on Image Synthesis , 2021, NeurIPS.

[26]  David J. Fleet,et al.  Image Super-Resolution via Iterative Refinement , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Valentin De Bortoli,et al.  Bayesian imaging using Plug & Play priors: when Langevin meets Tweedie , 2021, SIAM J. Imaging Sci..

[28]  Prafulla Dhariwal,et al.  Improved Denoising Diffusion Probabilistic Models , 2021, ICML.

[29]  Iain Murray,et al.  Maximum Likelihood Training of Score-Based Diffusion Models , 2021, NeurIPS.

[30]  P. Milanfar,et al.  Projected Distribution Loss for Image Enhancement , 2020, 2021 IEEE International Conference on Computational Photography (ICCP).

[31]  Abhishek Kumar,et al.  Score-Based Generative Modeling through Stochastic Differential Equations , 2020, ICLR.

[32]  K. Bouman,et al.  Deep Probabilistic Imaging: Uncertainty Quantification and Multi-modal Solution Characterization for Computational Imaging , 2020, AAAI.

[33]  Jiaming Song,et al.  Denoising Diffusion Implicit Models , 2020, ICLR.

[34]  Yi Wang,et al.  Extending LOUPE for K-space Under-sampling Pattern Optimization in Multi-coil MRI , 2020, MLMIR@MICCAI.

[35]  Pieter Abbeel,et al.  Denoising Diffusion Probabilistic Models , 2020, NeurIPS.

[36]  Andrew Gordon Wilson,et al.  Why Normalizing Flows Fail to Detect Out-of-Distribution Data , 2020, NeurIPS.

[37]  Yang Song,et al.  Generative Modeling by Estimating Gradients of the Data Distribution , 2019, NeurIPS.

[38]  Ali Ahmed,et al.  Invertible generative models for inverse problems: mitigating representation error and dataset bias , 2019, ICML.

[39]  Pieter Abbeel,et al.  Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design , 2019, ICML.

[40]  Jens Behrmann,et al.  Invertible Residual Networks , 2018, ICML.

[41]  Yee Whye Teh,et al.  Do Deep Generative Models Know What They Don't Know? , 2018, ICLR.

[42]  David Duvenaud,et al.  FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models , 2018, ICLR.

[43]  Prafulla Dhariwal,et al.  Glow: Generative Flow with Invertible 1x1 Convolutions , 2018, NeurIPS.

[44]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[45]  Thomas S. Huang,et al.  Free-Form Image Inpainting With Gated Convolution , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[46]  Sarthak Mittal,et al.  Diffusion Models , 2018, Financial Econometrics.

[47]  Kazunori Akiyama,et al.  Interferometric Imaging Directly with Closure Phases and Closure Amplitudes , 2018, 1803.07088.

[48]  S. Ikeda,et al.  Superresolution Interferometric Imaging with Sparse Modeling Using Total Squared Variation: Application to Imaging the Black Hole Shadow , 2018, 1802.05783.

[49]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[50]  Hiroshi Ishikawa,et al.  Globally and locally consistent image completion , 2017, ACM Trans. Graph..

[51]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[52]  Michael Elad,et al.  The Little Engine That Could: Regularization by Denoising (RED) , 2016, SIAM J. Imaging Sci..

[53]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[54]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Koray Kavukcuoglu,et al.  Pixel Recurrent Neural Networks , 2016, ICML.

[56]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[57]  Zoubin Ghahramani,et al.  Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference , 2015, ArXiv.

[58]  Surya Ganguli,et al.  Deep Unsupervised Learning using Nonequilibrium Thermodynamics , 2015, ICML.

[59]  Hugo Larochelle,et al.  MADE: Masked Autoencoder for Distribution Estimation , 2015, ICML.

[60]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[61]  Yoshua Bengio,et al.  NICE: Non-linear Independent Components Estimation , 2014, ICLR.

[62]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[63]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[64]  Brendt Wohlberg,et al.  Plug-and-Play priors for model based reconstruction , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[65]  Yair Weiss,et al.  "Natural Images, Gaussian Mixtures and Dead Leaves" , 2012, NIPS.

[66]  Yair Weiss,et al.  From learning models of natural image patches to whole image restoration , 2011, 2011 International Conference on Computer Vision.

[67]  Charalampos Tsitouras,et al.  Runge-Kutta pairs of order 5(4) satisfying only the first column simplifying assumption , 2011, Comput. Math. Appl..

[68]  Hugo Larochelle,et al.  The Neural Autoregressive Distribution Estimator , 2011, AISTATS.

[69]  Andrew Gelman,et al.  Handbook of Markov Chain Monte Carlo , 2011 .

[70]  E. Todeva Networks , 2007 .

[71]  Mariam Fraser,et al.  Event , 2006, Photographs and the Practice of History.

[72]  Erkki Oja,et al.  Independent component analysis: algorithms and applications , 2000, Neural Networks.

[73]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[74]  Ken D. Sauer,et al.  A generalized Gaussian image model for edge-preserving MAP estimation , 1993, IEEE Trans. Image Process..

[75]  B. Anderson Reverse-time diffusion equation models , 1982 .

[76]  J. Dormand,et al.  A family of embedded Runge-Kutta formulae , 1980 .

[77]  J. Kautz,et al.  Pseudoinverse-Guided Diffusion Models for Inverse Problems , 2023, ICLR.

[78]  Eero P. Simoncelli,et al.  Stochastic Solutions for Linear Inverse Problems using the Prior Implicit in a Denoiser , 2021, NeurIPS.

[79]  S. Panzeri,et al.  Synthesizing realistic neural population activity patterns using Generative Adversarial Networks , 2018, ICLR.

[80]  Jeff Hecht,et al.  Event horizon , 2011, Nature.

[81]  M. Hutchinson A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines , 1989 .

[82]  J. Skilling The Eigenvalues of Mega-dimensional Matrices , 1989 .