Bilinear Recovery Using Adaptive Vector-AMP

We consider the problem of jointly recovering the vector <inline-formula><tex-math notation="LaTeX">$\boldsymbol{b}$</tex-math></inline-formula> and the matrix <inline-formula><tex-math notation="LaTeX">$\boldsymbol{C}$</tex-math></inline-formula> from noisy measurements <inline-formula><tex-math notation="LaTeX">$\boldsymbol{Y} = \boldsymbol{A}(\boldsymbol{b})\boldsymbol{C} + \boldsymbol{W}$</tex-math></inline-formula>, where <inline-formula><tex-math notation="LaTeX">$\boldsymbol{A}(\cdot)$</tex-math></inline-formula> is a known affine linear function of <inline-formula><tex-math notation="LaTeX">$\boldsymbol{b}$</tex-math></inline-formula> (i.e., <inline-formula><tex-math notation="LaTeX">$\boldsymbol{A}(\boldsymbol{b})=\boldsymbol{A}_0+\sum _{i=1}^Q b_i \boldsymbol{A}_i$</tex-math></inline-formula> with known matrices <inline-formula><tex-math notation="LaTeX">$\boldsymbol{A}_i$</tex-math></inline-formula>). This problem has applications in matrix completion, robust PCA, dictionary learning, self-calibration, blind deconvolution, joint-channel/symbol estimation, compressive sensing with matrix uncertainty, and many other tasks. To solve this bilinear recovery problem, we propose the Bilinear Adaptive Vector Approximate Message Passing (VAMP) algorithm. We demonstrate numerically that the proposed approach is competitive with other state-of-the-art approaches to bilinear recovery, including lifted VAMP and Bilinear Generalized Approximate Message Passing.

[1]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[2]  Sundeep Rangan,et al.  Vector approximate message passing , 2017, 2017 IEEE International Symposium on Information Theory (ISIT).

[3]  Georgios B. Giannakis,et al.  Sparsity-Cognizant Total Least-Squares for Perturbed Compressive Sampling , 2010, IEEE Transactions on Signal Processing.

[4]  Ole Winther,et al.  Expectation Consistent Free Energies for Approximate Inference , 2004, NIPS.

[5]  Huan Wang,et al.  Exact Recovery of Sparsely-Used Dictionaries , 2012, COLT.

[6]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[7]  Chinmay Hegde,et al.  Sampling and Recovery of Pulse Streams , 2010, IEEE Transactions on Signal Processing.

[8]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[9]  Florent Krzakala,et al.  Phase Transitions and Sample Complexity in Bayes-Optimal Matrix Factorization , 2014, IEEE Transactions on Information Theory.

[10]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[11]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[12]  Sundeep Rangan,et al.  Expectation consistent approximate inference: Generalizations and convergence , 2016, 2016 IEEE International Symposium on Information Theory (ISIT).

[13]  Thomas Strohmer,et al.  Self-calibration and biconvex compressive sensing , 2015, ArXiv.

[14]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[15]  Richard G. Baraniuk,et al.  From Denoising to Compressed Sensing , 2014, IEEE Transactions on Information Theory.

[16]  Galen Reeves,et al.  The replica-symmetric prediction for compressed sensing with Gaussian matrices is exact , 2016, 2016 IEEE International Symposium on Information Theory (ISIT).

[17]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[18]  Martin J. Wainwright,et al.  Noisy matrix decomposition via convex relaxation: Optimal rates in high dimensions , 2011, ICML.

[19]  Volkan Cevher,et al.  Fixed Points of Generalized Approximate Message Passing With Arbitrary Matrices , 2016, IEEE Transactions on Information Theory.

[20]  Emmanuel J. Candès,et al.  Tight Oracle Inequalities for Low-Rank Matrix Recovery From a Minimal Number of Noisy Random Measurements , 2011, IEEE Transactions on Information Theory.

[21]  Philip Schniter,et al.  Expectation-Maximization Gaussian-Mixture Approximate Message Passing , 2012, IEEE Transactions on Signal Processing.

[22]  Toshiyuki Tanaka,et al.  Low-rank matrix reconstruction and clustering via approximate message passing , 2013, NIPS.

[23]  Volkan Cevher,et al.  Fixed Points of Generalized Approximate Message Passing With Arbitrary Matrices , 2013, IEEE Transactions on Information Theory.

[24]  G. Sapiro,et al.  A collaborative framework for 3D alignment and classification of heterogeneous subvolumes in cryo-electron tomography. , 2013, Journal of structural biology.

[25]  Yi Ma,et al.  The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices , 2010, Journal of structural biology.

[26]  T. Heskes,et al.  Expectation propagation for approximate inference in dynamic bayesian networks , 2002, UAI 2002.

[27]  M. Seeger.  Expectation Propagation for Exponential Families , 2005 .

[28]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[29]  Michael Elad,et al.  The Little Engine That Could: Regularization by Denoising (RED) , 2016, SIAM J. Imaging Sci..

[30]  Richard G. Baraniuk,et al.  A Field Guide to Forward-Backward Splitting with a FASTA Implementation , 2014, ArXiv.

[31]  Marc Teboulle,et al.  Fast Gradient-Based Algorithms for Constrained Total Variation Image Denoising and Deblurring Problems , 2009, IEEE Transactions on Image Processing.

[32]  Florent Krzakala,et al.  On convergence of approximate message passing , 2014, 2014 IEEE International Symposium on Information Theory.

[33]  Robert D. Nowak,et al.  Online identification and tracking of subspaces from highly incomplete information , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[34]  Nicolas Macris,et al.  The mutual information in random linear estimation , 2016, 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[35]  John Wright,et al.  Compressive principal component pursuit , 2012, 2012 IEEE International Symposium on Information Theory Proceedings.

[36]  Andrea Montanari,et al.  State Evolution for Approximate Message Passing with Non-Separable Functions , 2017, Information and Inference: A Journal of the IMA.

[37]  Volkan Cevher,et al.  Matrix Recipes for Hard Thresholding Methods , 2012, Journal of Mathematical Imaging and Vision.

[38]  Philip Schniter,et al.  Regularization by Denoising: Clarifications and New Interpretations , 2018, IEEE Transactions on Computational Imaging.

[39]  Lenka Zdeborová,et al.  Phase diagram of matrix compressed sensing , 2016, Physical review. E.

[40]  Sundeep Rangan,et al.  Generalized approximate message passing for estimation with random linear mixing , 2010, 2011 IEEE International Symposium on Information Theory Proceedings.

[41]  Justin K. Romberg,et al.  An Overview of Low-Rank Matrix Recovery From Incomplete Observations , 2016, IEEE Journal of Selected Topics in Signal Processing.

[42]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[43]  Léo Miolane.  Fundamental limits of low-rank matrix estimation , 2017, 1702.00473.

[44]  Aggelos K. Katsaggelos,et al.  Sparse Bayesian Methods for Low-Rank Matrix Estimation , 2011, IEEE Transactions on Signal Processing.

[45]  Ramji Venkataramanan,et al.  Finite-sample analysis of Approximate Message Passing , 2016, 2016 IEEE International Symposium on Information Theory (ISIT).

[46]  Laura Balzano,et al.  Incremental gradient on the Grassmannian for online foreground and background separation in subsampled video , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[47]  Li Ping,et al.  Orthogonal AMP , 2016, IEEE Access.

[48]  Andrea Montanari,et al.  The dynamics of message passing on dense graphs, with applications to compressed sensing , 2010, 2010 IEEE International Symposium on Information Theory.

[49]  Volkan Cevher,et al.  Bilinear Generalized Approximate Message Passing—Part II: Applications , 2014, IEEE Transactions on Signal Processing.

[50]  Sundeep Rangan,et al.  Hybrid Approximate Message Passing , 2011, IEEE Transactions on Signal Processing.

[51]  Emmanuel J. Candès,et al.  PhaseLift: Exact and Stable Signal Recovery from Magnitude Measurements via Convex Programming , 2011, ArXiv.

[52]  Emmanuel J. Candès,et al.  Matrix Completion With Noise , 2009, Proceedings of the IEEE.

[53]  Florent Krzakala,et al.  Constrained low-rank matrix estimation: phase transitions, approximate message passing and applications , 2017, ArXiv.

[54]  Rémi Gribonval,et al.  Convex Optimization Approaches for Blind Sensor Calibration Using Sparsity , 2013, IEEE Transactions on Signal Processing.

[55]  Philip Schniter,et al.  Parametric Bilinear Generalized Approximate Message Passing , 2015, IEEE Journal of Selected Topics in Signal Processing.

[56]  Shlomo Shamai,et al.  Support Recovery With Sparsely Sampled Free Random Matrices , 2011, IEEE Transactions on Information Theory.

[57]  Sundeep Rangan,et al.  Plug in estimation in high dimensional linear inverse problems a rigorous analysis , 2018, NeurIPS.

[58]  Guillermo Sapiro,et al.  Online Learning for Matrix Factorization and Sparse Coding , 2009, J. Mach. Learn. Res..

[59]  Sundeep Rangan,et al.  Rigorous Dynamics and Consistent Estimation in Arbitrarily Conditioned Linear Systems , 2017, NIPS.

[60]  Sundeep Rangan,et al.  Adaptive damping and mean removal for the generalized approximate message passing algorithm , 2014, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[61]  Galen Reeves.  Additivity of information in multilayer networks via additive Gaussian noise transforms , 2017, 2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[62]  Michael Elad,et al.  Dictionaries for Sparse Representation Modeling , 2010, Proceedings of the IEEE.

[63]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[64]  Volkan Cevher,et al.  Bilinear Generalized Approximate Message Passing—Part I: Derivation , 2013, IEEE Transactions on Signal Processing.

[65]  Matan Gavish,et al.  Near-optimal matrix recovery from random linear measurements , 2018, Proceedings of the National Academy of Sciences.

[66]  Michael Unser,et al.  Approximate Message Passing With Consistent Parameter Estimation and Applications to Sparse Learning , 2012, IEEE Transactions on Information Theory.

[67]  Yin Zhang,et al.  Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm , 2012, Mathematical Programming Computation.

[68]  Andrea Montanari,et al.  Universality in Polytope Phase Transitions and Message Passing Algorithms , 2012, ArXiv.

[69]  Keigo Takeuchi,et al.  Rigorous Dynamics of Expectation-Propagation-Based Signal Recovery from Unitarily Invariant Measurements , 2020, IEEE Transactions on Information Theory.

[70]  Justin K. Romberg,et al.  Blind Deconvolution Using Convex Programming , 2012, IEEE Transactions on Information Theory.

[71]  Ghassan Kawas Kaleh,et al.  Joint parameter estimation and symbol detection for linear or nonlinear unknown channels , 1994, IEEE Trans. Commun..

[72]  Aswin C. Sankaranarayanan,et al.  SpaRCS: Recovering low-rank and sparse matrices from compressive measurements , 2011, NIPS.