Scalable Plug-and-Play ADMM with Convergence Guarantees

Plug-and-play priors (PnP) is a broadly applicable methodology for solving inverse problems by exploiting statistical priors specified as denoisers. Recent work has reported the state-of-the-art performance of PnP algorithms using pre-trained deep neural nets as denoisers in a number of imaging applications. However, current PnP algorithms are impractical in large-scale settings due to their heavy computational and memory requirements. This work addresses this issue by proposing an incremental variant of the widely used PnP-ADMM algorithm, making it scalable to large-scale datasets. We theoretically analyze the convergence of the algorithm under a set of explicit assumptions, extending recent theoretical results in the area. Additionally, we show the effectiveness of our algorithm with nonsmooth data-fidelity terms and deep neural net priors, its fast convergence compared to existing PnP algorithms, and its scalability in terms of speed and memory.

[1]  Yanting Ma,et al.  Compressive Imaging via Approximate Message Passing With Image Denoising , 2014, IEEE Transactions on Signal Processing.

[2]  Alexandros G. Dimakis,et al.  Deep Learning Techniques for Inverse Problems in Imaging , 2020, IEEE Journal on Selected Areas in Information Theory.

[3]  Laura Waller,et al.  Physics-Based Learned Design: Optimized Coded-Illumination for Quantitative Phase Imaging , 2018, IEEE Transactions on Computational Imaging.

[4]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[5]  Arindam Banerjee,et al.  Online Alternating Direction Method , 2012, ICML.

[6]  Robert D. Nowak,et al.  An EM algorithm for wavelet-based image restoration , 2003, IEEE Trans. Image Process..

[7]  Michael Möller,et al.  Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  Matthias Zwicker,et al.  Deep Mean-Shift Priors for Image Restoration , 2017, NIPS.

[9]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[10]  Brendt Wohlberg,et al.  An Online Plug-and-Play Algorithm for Regularized Image Reconstruction , 2018, IEEE Transactions on Computational Imaging.

[11]  José M. Bioucas-Dias,et al.  Fast Image Recovery Using Variable Splitting and Constrained Optimization , 2009, IEEE Transactions on Image Processing.

[12]  Mário A. T. Figueiredo,et al.  Deconvolving Images With Unknown Boundaries Using the Alternating Direction Method of Multipliers , 2012, IEEE Transactions on Image Processing.

[13]  Michael Unser,et al.  Convolutional Neural Networks for Inverse Problems in Imaging: A Review , 2017, IEEE Signal Processing Magazine.

[14]  Volkan Cevher,et al.  Fast and Provable ADMM for Learning with Generative Priors , 2019, NeurIPS.

[15]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[16]  Richard G. Baraniuk,et al.  From Denoising to Compressed Sensing , 2014, IEEE Transactions on Information Theory.

[17]  Jeffrey A. Fessler,et al.  A Splitting-Based Iterative Algorithm for Accelerated Statistical X-Ray CT Reconstruction , 2012, IEEE Transactions on Medical Imaging.

[18]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[19]  Richard G. Baraniuk,et al.  BM3D-PRGAMP: Compressive phase retrieval based on BM3D denoising , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[20]  Wangmeng Zuo,et al.  Learning Deep CNN Denoiser Prior for Image Restoration , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Manfred Morari,et al.  Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks , 2019, NeurIPS.

[22]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[23]  Xiaohan Chen,et al.  Plug-and-Play Methods Provably Converge with Properly Trained Denoisers , 2019, ICML.

[24]  José M. Bioucas-Dias,et al.  A Convergent Image Fusion Algorithm Using Scene-Adapted Gaussian-Mixture-Based Denoising , 2019, IEEE Transactions on Image Processing.

[25]  Stephen P. Boyd,et al.  A Primer on Monotone Operator Methods , 2015 .

[26]  Ali Borji,et al.  CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research , 2015, ArXiv.

[27]  Sundeep Rangan,et al.  Plug in estimation in high dimensional linear inverse problems a rigorous analysis , 2018, NeurIPS.

[28]  s-taiji.  Dual Averaging and Proximal Gradient Descent for Online Alternating Direction Multiplier Method , 2013 .

[29]  Chinmay Hegde,et al.  Alternating Phase Projected Gradient Descent with Generative Priors for Solving Compressive Phase Retrieval , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[30]  Brendt Wohlberg,et al.  Plug-and-Play priors for model based reconstruction , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[31]  Laura Waller,et al.  Computational illumination for high-speed in vitro Fourier ptychographic microscopy , 2015, 1506.04274.

[32]  Charles A. Bouman,et al.  Plug-and-Play Methods for Magnetic Resonance Imaging: Using Denoisers for Image Recovery , 2019, IEEE Signal Processing Magazine.

[33]  Brendt Wohlberg,et al.  Efficient Algorithms for Convolutional Sparse Representations , 2016, IEEE Transactions on Image Processing.

[34]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[35]  Heng Huang,et al.  Faster Stochastic Alternating Direction Method of Multipliers for Nonconvex Optimization , 2020, ICML.

[36]  Alexander G. Gray,et al.  Stochastic Alternating Direction Method of Multipliers , 2013, ICML.

[37]  Aggelos K. Katsaggelos,et al.  Using Deep Neural Networks for Inverse Problems in Imaging: Beyond Analytical Methods , 2018, IEEE Signal Processing Magazine.

[38]  Lei Zhang,et al.  Deep Plug-And-Play Super-Resolution for Arbitrary Blur Kernels , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Ying Fu,et al.  Tuning-free Plug-and-Play Proximal Algorithm for Inverse Imaging Problems , 2020, ICML.

[40]  Charles A. Bouman,et al.  Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolation , 2015, IEEE Transactions on Computational Imaging.

[41]  Dimitri P. Bertsekas,et al.  Incremental proximal methods for large scale convex optimization , 2011, Math. Program..

[42]  Yaoliang Yu,et al.  Better Approximation and Faster Algorithm Using the Proximal Average , 2013, NIPS.

[43]  Michael K. Ng,et al.  Solving Constrained Total-variation Image Restoration and Reconstruction Problems via Alternating Direction Methods , 2010, SIAM J. Sci. Comput..

[44]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[45]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[46]  Michael Elad,et al.  DeepRED: Deep Image Prior Powered by RED , 2019, ICCV 2019.

[47]  Raja Giryes,et al.  Image Restoration by Iterative Denoising and Backward Projections , 2017, IEEE Transactions on Image Processing.

[48]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[49]  Chinmay Hegde,et al.  Solving Linear Inverse Problems Using Gan Priors: An Algorithm with Provable Guarantees , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[50]  Charles A. Bouman,et al.  Plug-and-Play Unplugged: Optimization Free Reconstruction using Consensus Equilibrium , 2017, SIAM J. Imaging Sci..

[51]  Leon Wenliang Zhong,et al.  Fast Stochastic Alternating Direction Method of Multipliers , 2013, ICML.

[52]  Heinz H. Bauschke,et al.  The Proximal Average: Basic Theory , 2008, SIAM J. Optim..

[53]  M. Nikolova.  An Algorithm for Total Variation Minimization and Applications , 2004 .

[54]  Michael Elad,et al.  The Little Engine That Could: Regularization by Denoising (RED) , 2016, SIAM J. Imaging Sci..

[55]  Antonin Chambolle,et al.  A l1-Unified Variational Framework for Image Restoration , 2004, ECCV.

[56]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[57]  Yu Sun,et al.  Block Coordinate Regularization by Denoising , 2019, IEEE Transactions on Computational Imaging.

[58]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[59]  Audrey Repetti,et al.  Building Firmly Nonexpansive Convolutional Neural Networks , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[60]  Philip M. Long,et al.  The Singular Values of Convolutional Layers , 2018, ICLR.

[61]  Marc Teboulle,et al.  Fast Gradient-Based Algorithms for Constrained Total Variation Image Denoising and Deblurring Problems , 2009, IEEE Transactions on Image Processing.

[62]  Stanley H. Chan,et al.  Plug-and-Play ADMM for Image Restoration: Fixed-Point Convergence and Applications , 2016, IEEE Transactions on Computational Imaging.

[63]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[64]  Steen Moeller,et al.  Deep-Learning Methods for Parallel Magnetic Resonance Imaging Reconstruction: A Survey of the Current Approaches, Trends, and Issues , 2020, IEEE Signal Processing Magazine.

[65]  Yuqi Li,et al.  GAN-based Projector for Faster Recovery in Compressed Sensing with Convergence Guarantees , 2019, ArXiv.

[66]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[67]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[68]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[69]  Lei Tian,et al.  High-throughput intensity diffraction tomography with a computational microscope. , 2018, Biomedical optics express.

[70]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[71]  Guangming Shi,et al.  Denoising Prior Driven Deep Neural Network for Image Restoration , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.