Exploiting the structure effectively and efficiently in low rank matrix recovery

Low rank model arises from a wide range of applications, including machine learning, signal processing, computer algebra, computer vision, and imaging science. Low rank matrix recovery is about reconstructing a low rank matrix from incomplete measurements. In this survey we review recent developments on low rank matrix recovery, focusing on three typical scenarios: matrix sensing, matrix completion and phase retrieval. An overview of effective and efficient approaches for the problem is given, including nuclear norm minimization, projected gradient descent based on matrix factorization, and Riemannian optimization based on the embedded manifold of low rank matrices. Numerical recipes of different approaches are emphasized while accompanied by the corresponding theoretical recovery guarantees.

[1]  John Wright,et al.  A Geometric Analysis of Phase Retrieval , 2016, International Symposium on Information Theory.

[2]  O. Bunk,et al.  Diffractive imaging for periodic samples: retrieving one-dimensional concentration profiles across microfluidic channels. , 2007, Acta crystallographica. Section A, Foundations of crystallography.

[3]  Yudong Chen,et al.  Incoherence-Optimal Matrix Completion , 2013, IEEE Transactions on Information Theory.

[4]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[5]  D. Botstein,et al.  Singular value decomposition for genome-wide expression data processing and modeling. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[6]  Sujay Sanghavi,et al.  Completing any low-rank matrix, provably , 2013, J. Mach. Learn. Res..

[7]  Jared Tanner,et al.  Normalized Iterative Hard Thresholding for Matrix Completion , 2013, SIAM J. Sci. Comput..

[8]  Yair Carmon,et al.  Accelerated Methods for Non-Convex Optimization , 2016, SIAM J. Optim..

[9]  Yuxin Chen,et al.  Robust Spectral Compressed Sensing via Structured Matrix Completion , 2013, IEEE Transactions on Information Theory.

[10]  Douglas Lanman,et al.  Content-adaptive parallax barriers: optimizing dual-layer 3D displays using low-rank light field factorization , 2010, ACM Trans. Graph..

[11]  Prateek Jain,et al.  Non-convex Robust PCA , 2014, NIPS.

[12]  Tengyu Ma,et al.  Finding Approximate Local Minima for Nonconvex Optimization in Linear Time , 2016, ArXiv.

[13]  Emmanuel J. Candès,et al.  Tight Oracle Inequalities for Low-Rank Matrix Recovery From a Minimal Number of Noisy Random Measurements , 2011, IEEE Transactions on Information Theory.

[14]  Emmanuel J. Candès,et al.  PhaseLift: Exact and Stable Signal Recovery from Magnitude Measurements via Convex Programming , 2011, ArXiv.

[15]  Thomas Strohmer,et al.  Painless Breakups—Efficient Demixing of Low Rank Matrices , 2017, ArXiv.

[16]  G. Sapiro,et al.  A collaborative framework for 3D alignment and classification of heterogeneous subvolumes in cryo-electron tomography. , 2013, Journal of structural biology.

[17]  J. Tanner,et al.  Low rank matrix completion by alternating steepest descent methods , 2016 .

[18]  Tengyu Ma,et al.  Finding approximate local minima faster than gradient descent , 2016, STOC.

[19]  Xiaoming Yuan,et al.  Recovering Low-Rank and Sparse Components of Matrices from Incomplete and Noisy Observations , 2011, SIAM J. Optim..

[20]  Xiaodong Li,et al.  Solving Quadratic Equations via PhaseLift When There Are About as Many Equations as Unknowns , 2012, Found. Comput. Math..

[21]  Yair Carmon,et al.  Accelerated Methods for NonConvex Optimization , 2018, SIAM J. Optim..

[22]  Yi Zheng,et al.  No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis , 2017, ICML.

[23]  J. Ragot,et al.  Fault detection and isolation with robust principal component analysis , 2008, MED 2008.

[24]  Yin Zhang,et al.  Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm , 2012, Mathematical Programming Computation.

[25]  Yingbin Liang,et al.  Provable Non-convex Phase Retrieval with Outliers: Median TruncatedWirtinger Flow , 2016, ICML.

[26]  Robert E. Mahony,et al.  Optimization Algorithms on Matrix Manifolds , 2007 .

[27]  Yi Ma,et al.  The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices , 2010, Journal of structural biology.

[28]  Zuowei Shen,et al.  Robust Video Restoration by Joint Sparse and Low Rank Matrix Approximation , 2011, SIAM J. Imaging Sci..

[29]  Thomas Strohmer,et al.  Blind Deconvolution Meets Blind Demixing: Algorithms and Performance Bounds , 2015, IEEE Transactions on Information Theory.

[30]  Mike E. Davies,et al.  Iterative Hard Thresholding for Compressed Sensing , 2008, ArXiv.

[31]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[32]  Moritz Hardt,et al.  Understanding Alternating Minimization for Matrix Completion , 2013, 2014 IEEE 55th Annual Symposium on Foundations of Computer Science.

[33]  Martin J. Wainwright,et al.  Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees , 2015, ArXiv.

[34]  Pierre-Antoine Absil,et al.  RTRMC: A Riemannian trust-region method for low-rank matrix completion , 2011, NIPS.

[35]  Justin K. Romberg,et al.  Blind Deconvolution Using Convex Programming , 2012, IEEE Transactions on Information Theory.

[36]  Wen Huang,et al.  Solving PhaseLift by Low-rank Riemannian Optimization Methods , 2016, ICCS.

[37]  Andrea J. Goldsmith,et al.  Exact and Stable Covariance Estimation From Quadratic Sampling via Convex Programming , 2013, IEEE Transactions on Information Theory.

[38]  D. Donoho,et al.  Sparse MRI: The application of compressed sensing for rapid MR imaging , 2007, Magnetic resonance in medicine.

[39]  Martin J. Wainwright,et al.  Restricted strong convexity and weighted matrix completion: Optimal bounds with noise , 2010, J. Mach. Learn. Res..

[40]  Qi Tian,et al.  Statistical modeling of complex backgrounds for foreground object detection , 2004, IEEE Transactions on Image Processing.

[41]  Bamdev Mishra,et al.  R3MC: A Riemannian three-factor algorithm for low-rank matrix completion , 2013, 53rd IEEE Conference on Decision and Control.

[42]  Bart Vandereycken,et al.  Low-Rank Matrix Completion by Riemannian Optimization , 2013, SIAM J. Optim..

[43]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[44]  Douglas B. Terry,et al.  Using collaborative filtering to weave an information tapestry , 1992, CACM.

[45]  Yousef Saad,et al.  Scaled Gradients on Grassmann Manifolds for Matrix Completion , 2012, NIPS.

[46]  Xiaodong Li,et al.  Rapid, Robust, and Reliable Blind Deconvolution via Nonconvex Optimization , 2016, Applied and Computational Harmonic Analysis.

[47]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2009, Found. Comput. Math..

[48]  Jeffrey D. Blanchard,et al.  CGIHT: Conjugate Gradient Iterative Hard Thresholding for Compressed Sensing and Matrix Completion , 2015 .

[49]  Jian-Feng Cai,et al.  Accelerated NMR spectroscopy with low-rank reconstruction. , 2015, Angewandte Chemie.

[50]  Jian-Feng Cai,et al.  Fast and Provable Algorithms for Spectrally Sparse Signal Reconstruction via Low-Rank Hankel Matrix Completion , 2016, Applied and Computational Harmonic Analysis.

[51]  Jens Eisert,et al.  Guaranteed recovery of quantum processes from few measurements , 2017, Quantum.

[52]  Michael I. Jordan,et al.  How to Escape Saddle Points Efficiently , 2017, ICML.

[53]  David Gross,et al.  Recovering Low-Rank Matrices From Few Coefficients in Any Basis , 2009, IEEE Transactions on Information Theory.

[54]  Stephen P. Boyd,et al.  Log-det heuristic for matrix rank minimization with applications to Hankel and Euclidean distance matrices , 2003, Proceedings of the 2003 American Control Conference, 2003..

[55]  David L Donoho,et al.  Compressed sensing , 2006, IEEE Transactions on Information Theory.

[56]  John D. Lafferty,et al.  Convergence Analysis for Rectangular Matrix Completion Using Burer-Monteiro Factorization and Gradient Descent , 2016, ArXiv.

[57]  Max Simchowitz,et al.  Low-rank Solutions of Linear Matrix Equations via Procrustes Flow , 2015, ICML.

[58]  Justin K. Romberg,et al.  An Overview of Low-Rank Matrix Recovery From Incomplete Observations , 2016, IEEE Journal of Selected Topics in Signal Processing.

[59]  H. Leonhardt,et al.  A guide to super-resolution fluorescence microscopy , 2010, The Journal of cell biology.

[60]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[61]  Yudong Chen,et al.  Harnessing Structures in Big Data via Guaranteed Low-Rank Matrix Estimation: Recent Theory and Fast Algorithms via Convex and Nonconvex Optimization , 2018, IEEE Signal Processing Magazine.

[62]  Yi-Kai Liu,et al.  Universal low-rank matrix recovery from Pauli measurements , 2011, NIPS.

[63]  Shiqian Ma,et al.  Fixed point and Bregman iterative methods for matrix rank minimization , 2009, Math. Program..

[64]  Martin Jaggi,et al.  A Simple Algorithm for Nuclear Norm Regularized Problems , 2010, ICML.

[65]  Bamdev Mishra,et al.  Fixed-rank matrix factorizations and Riemannian low-rank optimization , 2012, Comput. Stat..

[66]  Constantine Caramanis,et al.  Fast Algorithms for Robust PCA via Gradient Descent , 2016, NIPS.

[67]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[68]  Hui Zhang,et al.  Strongly Convex Programming for Exact Matrix Completion and Robust Principal Component Analysis , 2011, ArXiv.

[69]  Yudong Chen,et al.  Leave-One-Out Approach for Matrix Completion: Primal and Dual Analysis , 2018, IEEE Transactions on Information Theory.

[70]  Tony F. Chan,et al.  Guarantees of Riemannian Optimization for Low Rank Matrix Recovery , 2015, SIAM J. Matrix Anal. Appl..

[71]  Joel A. Tropp,et al.  Living on the edge: phase transitions in convex programs with random data , 2013, 1303.6672.

[72]  V. Koltchinskii,et al.  Nuclear norm penalization and optimal rates for noisy low rank matrix completion , 2010, 1011.6256.

[73]  Andrea Montanari,et al.  Matrix completion from a few entries , 2009, ISIT.

[74]  M. Viberg,et al.  Two decades of array signal processing research: the parametric approach , 1996, IEEE Signal Process. Mag..

[75]  Sujay Sanghavi,et al.  Clustering Sparse Graphs , 2012, NIPS.

[76]  Thomas Strohmer,et al.  Self-calibration and biconvex compressive sensing , 2015, ArXiv.

[77]  Inderjit S. Dhillon,et al.  Guaranteed Rank Minimization via Singular Value Projection , 2009, NIPS.

[78]  Yong-Jin Liu,et al.  An implementable proximal point algorithmic framework for nuclear norm minimization , 2012, Math. Program..

[79]  Adi Shraibman,et al.  Rank, Trace-Norm and Max-Norm , 2005, COLT.

[80]  Bamdev Mishra,et al.  A Riemannian geometry for low-rank matrix completion , 2012, ArXiv.

[81]  Emmanuel J. Candès,et al.  The Power of Convex Relaxation: Near-Optimal Matrix Completion , 2009, IEEE Transactions on Information Theory.

[82]  Prateek Jain,et al.  Low-rank matrix completion using alternating minimization , 2012, STOC '13.

[83]  Yonina C. Eldar,et al.  Solving Systems of Random Quadratic Equations via Truncated Amplitude Flow , 2016, IEEE Transactions on Information Theory.

[84]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[85]  Jian-Feng Cai,et al.  Spectral Compressed Sensing via Projected Gradient Descent , 2017, SIAM J. Optim..

[86]  J. Miao,et al.  Extending X-ray crystallography to allow the imaging of noncrystalline materials, cells, and single protein complexes. , 2008, Annual review of physical chemistry.

[87]  T. Chan,et al.  Guarantees of riemannian optimization for low rank matrix completion , 2016, Inverse Problems & Imaging.

[88]  Jared Tanner.  Efficient algorithms for compressed sensing and matrix completion , 2014 .

[89]  Paris Smaragdis,et al.  Singing-voice separation from monaural recordings using robust principal component analysis , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[90]  John Wright,et al.  Scalable Robust Matrix Recovery: Frank-Wolfe Meets Proximal Methods , 2014, SIAM J. Sci. Comput..

[91]  Zhiqiang Xu,et al.  The minimal measurement number for low-rank matrices recovery , 2015, ArXiv.

[92]  Jian-Feng Cai,et al.  Solving Systems of Phaseless Equations Via Riemannian Optimization with Optimal Sampling Complexity , 2018, Journal of Computational Mathematics.

[93]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[94]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[95]  Emre Ertin,et al.  Sparsity and Compressed Sensing in Radar Imaging , 2010, Proceedings of the IEEE.

[96]  Xiaoming Yuan,et al.  Matrix completion via an alternating direction method , 2012 .

[97]  Kim-Chuan Toh,et al.  SDPT3 -- A Matlab Software Package for Semidefinite Programming , 1996 .

[98]  S. Yun,et al.  An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems , 2009 .

[99]  Ewout van den Berg,et al.  1-Bit Matrix Completion , 2012, ArXiv.

[100]  Hossein Mobahi,et al.  Holistic 3D reconstruction of urban structures from low-rank textures , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[101]  Hyunsoo Kim,et al.  Sparse Non-negative Matrix Factorizations via Alternating Non-negativity-constrained Least Squares , 2006 .

[102]  Zhi-Quan Luo,et al.  Guaranteed Matrix Completion via Non-Convex Factorization , 2014, IEEE Transactions on Information Theory.

[103]  Jian-Feng Cai,et al.  Accelerated Alternating Projections for Robust Principal Component Analysis , 2017, J. Mach. Learn. Res..

[104]  Xiaodong Li,et al.  Phase Retrieval via Wirtinger Flow: Theory and Algorithms , 2014, IEEE Transactions on Information Theory.

[105]  Emmanuel J. Candès,et al.  Tight oracle bounds for low-rank matrix recovery from a minimal number of random measurements , 2010, ArXiv.

[106]  S. Yun,et al.  An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems , 2009 .

[107]  Yuxin Chen,et al.  Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systems , 2015, NIPS.

[108]  Benjamin Recht,et al.  A Simpler Approach to Matrix Completion , 2009, J. Mach. Learn. Res..

[109]  Furong Huang,et al.  Escaping From Saddle Points - Online Stochastic Gradient for Tensor Decomposition , 2015, COLT.

[110]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[111]  Holger Rauhut,et al.  Low rank matrix recovery from rank one measurements , 2014, ArXiv.

[112]  Pablo A. Parrilo,et al.  The Convex Geometry of Linear Inverse Problems , 2010, Foundations of Computational Mathematics.

[113]  Robert W. Harrison,et al.  Phase problem in crystallography , 1993 .

[114]  Jian-Feng Cai,et al.  Data recovery on a manifold from linear samples: theory and computation , 2018 .