Plug in estimation in high dimensional linear inverse problems a rigorous analysis

Estimating a vector $\mathbf{x}$ from noisy linear measurements $\mathbf{Ax+w}$ often requires use of prior knowledge or structural constraints on $\mathbf{x}$ for accurate reconstruction. Several recent works have considered combining linear least-squares estimation with a generic or plug-in ``denoiser" function that can be designed in a modular manner based on the prior knowledge about $\mathbf{x}$. While these methods have shown excellent performance, it has been difficult to obtain rigorous performance guarantees. This work considers plug-in denoising combined with the recently-developed Vector Approximate Message Passing (VAMP) algorithm, which is itself derived via Expectation Propagation techniques. It shown that the mean squared error of this ``plug-in" VAMP can be exactly predicted for a large class of high-dimensional random $\Abf$ and denoisers. The method is illustrated in image reconstruction and parametric bilinear estimation.

[1]  Brendt Wohlberg,et al.  Plug-and-Play priors for model based reconstruction , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[2]  Volkan Cevher,et al.  Fixed Points of Generalized Approximate Message Passing With Arbitrary Matrices , 2016, IEEE Transactions on Information Theory.

[3]  Andrea Montanari,et al.  Accurate Prediction of Phase Transitions in Compressed Sensing via a Connection to Minimax Denoising , 2011, IEEE Transactions on Information Theory.

[4]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[5]  Emmanuel J. Candès,et al.  Unbiased Risk Estimates for Singular Value Thresholding and Spectral Estimators , 2012, IEEE Transactions on Signal Processing.

[6]  Florent Krzakala,et al.  Phase transitions in sparse PCA , 2015, 2015 IEEE International Symposium on Information Theory (ISIT).

[7]  Yanting Ma,et al.  Analysis of approximate message passing with a class of non-separable denoisers , 2017, 2017 IEEE International Symposium on Information Theory (ISIT).

[8]  Ce Liu,et al.  Deep Convolutional Neural Network for Image Deconvolution , 2014, NIPS.

[9]  Sundeep Rangan,et al.  Denoising-based Vector AMP , 2016 .

[10]  Toshiyuki Tanaka,et al.  Low-rank matrix reconstruction and clustering via approximate message passing , 2013, NIPS.

[11]  Ole Winther,et al.  Bayesian Inference for Structured Spike and Slab Priors , 2014, NIPS.

[12]  Shiliang Sun,et al.  Multitask Twin Support Vector Machines , 2012, ICONIP.

[13]  Sundeep Rangan,et al.  Iterative estimation of constrained rank-one matrices in noise , 2012, 2012 IEEE International Symposium on Information Theory Proceedings.

[14]  Sundeep Rangan,et al.  Generalized approximate message passing for estimation with random linear mixing , 2010, 2011 IEEE International Symposium on Information Theory Proceedings.

[15]  Hongqiang Wang,et al.  BM3D vector approximate message passing for radar coded-aperture imaging , 2017, 2017 Progress in Electromagnetics Research Symposium - Fall (PIERS - FALL).

[16]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[17]  Ole Winther,et al.  Expectation Consistent Approximate Inference , 2005, J. Mach. Learn. Res..

[18]  Phillipp Meister,et al.  Statistical Signal Processing Detection Estimation And Time Series Analysis , 2016 .

[19]  Sundeep Rangan,et al.  Denoising based Vector Approximate Message Passing , 2016, ArXiv.

[20]  Sundeep Rangan,et al.  On the convergence of approximate message passing with arbitrary matrices , 2014, 2014 IEEE International Symposium on Information Theory.

[21]  Adel Javanmard,et al.  State Evolution for General Approximate Message Passing Algorithms, with Applications to Spatial Coupling , 2012, ArXiv.

[22]  Stanley H. Chan,et al.  Parameter-free Plug-and-Play ADMM for image restoration , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[23]  Florent Krzakala,et al.  On convergence of approximate message passing , 2014, 2014 IEEE International Symposium on Information Theory.

[24]  Enhong Chen,et al.  Image Denoising and Inpainting with Deep Neural Networks , 2012, NIPS.

[25]  Sundeep Rangan,et al.  Vector approximate message passing , 2017, 2017 IEEE International Symposium on Information Theory (ISIT).

[26]  Georgios B. Giannakis,et al.  Sparsity-Cognizant Total Least-Squares for Perturbed Compressive Sampling , 2010, IEEE Transactions on Signal Processing.

[27]  Philip Schniter,et al.  Joint Channel-Estimation and Equalization of Single-Carrier Systems via Bilinear AMP , 2018, IEEE Transactions on Signal Processing.

[28]  Philip Schniter,et al.  Parametric Bilinear Generalized Approximate Message Passing , 2015, IEEE Journal of Selected Topics in Signal Processing.

[29]  Sundeep Rangan,et al.  AMP-Inspired Deep Networks for Sparse Linear Inverse Problems , 2016, IEEE Transactions on Signal Processing.

[30]  Li Ping,et al.  Orthogonal AMP , 2016, IEEE Access.

[31]  Emmanuel J. Candès,et al.  PhaseLift: Exact and Stable Signal Recovery from Magnitude Measurements via Convex Programming , 2011, ArXiv.

[32]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[33]  C. Stein A bound for the error in the normal approximation to the distribution of a sum of dependent random variables , 1972 .

[34]  Andrea Montanari,et al.  Information-theoretically optimal sparse PCA , 2014, 2014 IEEE International Symposium on Information Theory.

[35]  Sundeep Rangan,et al.  Rigorous Dynamics and Consistent Estimation in Arbitrarily Conditioned Linear Systems , 2017, NIPS.

[36]  Sundeep Rangan,et al.  Adaptive damping and mean removal for the generalized approximate message passing algorithm , 2014, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[37]  Michael Unser,et al.  Approximate Message Passing With Consistent Parameter Estimation and Applications to Sparse Learning , 2012, IEEE Transactions on Information Theory.

[38]  Tom E. Bishop,et al.  Blind Deconvolution , 2014, Computer Vision, A Reference Guide.

[39]  Keigo Takeuchi,et al.  Rigorous Dynamics of Expectation-Propagation-Based Signal Recovery from Unitarily Invariant Measurements , 2020, IEEE Transactions on Information Theory.

[40]  Justin K. Romberg,et al.  Blind Deconvolution Using Convex Programming , 2012, IEEE Transactions on Information Theory.

[41]  Andrea Montanari,et al.  State Evolution for Approximate Message Passing with Non-Separable Functions , 2017, Information and Inference: A Journal of the IMA.

[42]  Justin K. Romberg,et al.  An Overview of Low-Rank Matrix Recovery From Incomplete Observations , 2016, IEEE Journal of Selected Topics in Signal Processing.

[43]  Thomas Strohmer,et al.  Self-calibration and biconvex compressive sensing , 2015, ArXiv.

[44]  Andrea Montanari,et al.  The dynamics of message passing on dense graphs, with applications to compressed sensing , 2010, 2010 IEEE International Symposium on Information Theory.

[45]  Sundeep Rangan,et al.  Hybrid Approximate Message Passing , 2011, IEEE Transactions on Signal Processing.

[46]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[47]  Sundeep Rangan,et al.  Expectation consistent approximate inference: Generalizations and convergence , 2016, 2016 IEEE International Symposium on Information Theory (ISIT).

[48]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[49]  Armeen Taeb,et al.  Maximin Analysis of Message Passing Algorithms for Recovering Block Sparse Signals , 2013, ArXiv.

[50]  Richard G. Baraniuk,et al.  From Denoising to Compressed Sensing , 2014, IEEE Transactions on Information Theory.