Inverse problem regularization with hierarchical variational autoencoders

In this paper, we propose to regularize ill-posed inverse problems using a deep hierarchical variational autoencoder (HVAE) as an image prior. The proposed method synthesizes the advantages of i) denoiser-based Plug \&Play approaches and ii) generative model based approaches to inverse problems. First, we exploit VAE properties to design an efficient algorithm that benefits from convergence guarantees of Plug-and-Play (PnP) methods. Second, our approach is not restricted to specialized datasets and the proposed PnP-HVAE model is able to solve image restoration problems on natural images of any size. Our experiments show that the proposed PnP-HVAE method is competitive with both SOTA denoiser-based PnP approaches, and other SOTA restoration methods based on generative models.

[1]  David J. Fleet,et al.  Image Super-Resolution via Iterative Refinement , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Y. Kabashima,et al.  Diffusion Model Based Posterior Sampling for Noisy Linear Inverse Problems , 2022, ArXiv.

[3]  Troy Luhman,et al.  Optimizing Hierarchical Image VAEs for Sample Quality , 2022, ArXiv.

[4]  Michael T. McCann,et al.  Diffusion Posterior Sampling for General Noisy Inverse Problems , 2022, ICLR.

[5]  Rayhane Mama,et al.  Efficient-VDVAE: Less is more , 2022, ArXiv.

[6]  Michael Elad,et al.  Denoising Diffusion Restoration Models , 2022, NeurIPS.

[7]  L. Gool,et al.  RePaint: Inpainting using Denoising Diffusion Probabilistic Models , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  S. Ermon,et al.  Solving Inverse Problems in Medical Imaging with Score-Based Generative Models , 2021, ICLR.

[9]  David J. Fleet,et al.  Palette: Image-to-Image Diffusion Models , 2021, SIGGRAPH.

[10]  N. Papadakis,et al.  Gradient Step Denoiser for convergent Plug-and-Play , 2021, ICLR.

[11]  P. Milanfar,et al.  Interpretable Unsupervised Diversity Denoising and Artefact Removal , 2021, ICLR.

[12]  Andrés Almansa,et al.  Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior , 2021, SIAM J. Imaging Sci..

[13]  Luc Van Gool,et al.  Plug-and-Play Image Restoration With Deep Denoiser Prior , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Bo Dai,et al.  Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  G. Steidl,et al.  PatchNR: Learning from Small Data by Patch Normalizing Flow Regularization , 2022, ArXiv.

[16]  Prafulla Dhariwal,et al.  Diffusion Models Beat GANs on Image Synthesis , 2021, NeurIPS.

[17]  Jakob Drachmann Havtorn,et al.  Hierarchical VAEs Know What They Don't Know , 2021, ICML.

[18]  Alexandros G. Dimakis,et al.  Intermediate Layer Optimization for Inverse Problems using Deep Generative Models , 2021, ICML.

[19]  Nicolas Papadakis,et al.  Learning local regularization for variational image restoration , 2021, SSVM.

[20]  Thomas Oberlin,et al.  Regularization via Deep Generative Models: an Analysis Point of View , 2021, 2021 IEEE International Conference on Image Processing (ICIP).

[21]  Jean-Christophe Pesquet,et al.  Learning Maximally Monotone Operators for Image Recovery , 2020, SIAM J. Imaging Sci..

[22]  Abhishek Kumar,et al.  Score-Based Generative Modeling through Stochastic Differential Equations , 2020, ICLR.

[23]  Rewon Child,et al.  Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images , 2020, ICLR.

[24]  Jan Kautz,et al.  NVAE: A Deep Hierarchical Variational Autoencoder , 2020, NeurIPS.

[25]  Pieter Abbeel,et al.  Denoising Diffusion Probabilistic Models , 2020, NeurIPS.

[26]  C. Rudin,et al.  PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Tero Karras,et al.  Analyzing and Improving the Image Quality of StyleGAN , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Volkan Cevher,et al.  Fast and Provable ADMM for Learning with Generative Priors , 2019, NeurIPS.

[29]  Xiaohan Chen,et al.  Plug-and-Play Methods Provably Converge with Properly Trained Denoisers , 2019, ICML.

[30]  Yuqi Li,et al.  GAN-Based Projector for Faster Recovery With Convergence Guarantees in Linear Inverse Problems , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[31]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[33]  Reinhard Heckel,et al.  A Provably Convergent Scheme for Compressive Sensing Under Random Generative Priors , 2018, Journal of Fourier Analysis and Applications.

[34]  Vladislav Voroninski,et al.  Phase Retrieval Under a Generative Prior , 2018, NeurIPS.

[35]  Prafulla Dhariwal,et al.  Glow: Generative Flow with Invertible 1x1 Convolutions , 2018, NeurIPS.

[36]  Chinmay Hegde,et al.  Solving Linear Inverse Problems Using Gan Priors: An Algorithm with Provable Guarantees , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[37]  Eirikur Agustsson,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[38]  Stefano Ermon,et al.  Learning Hierarchical Features from Deep Generative Models , 2017, ICML.

[39]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[40]  Alexandros G. Dimakis,et al.  Compressed Sensing using Generative Models , 2017, ICML.

[41]  Michael Elad,et al.  The Little Engine That Could: Regularization by Denoising (RED) , 2016, SIAM J. Imaging Sci..

[42]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[43]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[44]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[45]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[46]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[47]  Brendt Wohlberg,et al.  Plug-and-Play priors for model based reconstruction , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[48]  Yair Weiss,et al.  From learning models of natural image patches to whole image restoration , 2011, 2011 International Conference on Computer Vision.

[49]  Hédy Attouch,et al.  Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality , 2008, Math. Oper. Res..

[50]  Frédo Durand,et al.  Understanding and evaluating blind deconvolution algorithms , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[51]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.