Rank 2r Iterative Least Squares: Efficient Recovery of Ill-Conditioned Low Rank Matrices from Few Entries

We present a new, simple and computationally efficient iterative method for low rank matrix completion. Our method is inspired by the class of factorization-type iterative algorithms, but substantially differs from them in the way the problem is cast. Precisely, given a target rank $r$, instead of optimizing on the manifold of rank $r$ matrices, we allow our interim estimated matrix to have a specific over-parametrized rank $2r$ structure. Our algorithm, denoted \texttt{R2RILS}, for rank $2r$ iterative least squares, thus has low memory requirements, and at each iteration it solves a computationally cheap sparse least-squares problem. We motivate our algorithm by its theoretical analysis for the simplified case of a rank-1 matrix. Empirically, \texttt{R2RILS} is able to recover, with machine precision, ill conditioned low rank matrices from very few observations -- near the information limit. Finally, \texttt{R2RILS} is stable to corruption of the observed entries by additive zero mean Gaussian noise.

[1]  Stephen P. Boyd,et al.  A rank minimization heuristic with application to minimum order system approximation , 2001, Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148).

[2]  Volkan Cevher,et al.  Matrix Recipes for Hard Thresholding Methods , 2012, Journal of Mathematical Imaging and Vision.

[3]  Andrew W. Fitzgibbon,et al.  Secrets of Matrix Factorization: Approximations, Numerics, Manifold Optimization and Random Restarts , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[4]  Robert Tibshirani,et al.  Spectral Regularization Algorithms for Learning Large Incomplete Matrices , 2010, J. Mach. Learn. Res..

[5]  Zhi-Quan Luo,et al.  Guaranteed Matrix Completion via Non-Convex Factorization , 2014, IEEE Transactions on Information Theory.

[6]  Robert D. Nowak,et al.  Online identification and tracking of subspaces from highly incomplete information , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[7]  Nathan Srebro,et al.  Fast maximum margin matrix factorization for collaborative prediction , 2005, ICML.

[8]  Andrea Montanari,et al.  Matrix completion from a few entries , 2009, 2009 IEEE International Symposium on Information Theory.

[9]  Prateek Jain,et al.  Fast Exact Matrix Completion with Finite Samples , 2014, COLT.

[10]  Tianxi Li,et al.  Matrix completion from a computational statistics perspective , 2019, WIREs Computational Statistics.

[11]  Ken Hayami.  Convergence of the Conjugate Gradient Method on Singular Systems , 2018 .

[12]  Goran Marjanovic,et al.  On $l_q$ Optimization and Matrix Completion , 2012, IEEE Transactions on Signal Processing.

[13]  Nigel Boston,et al.  A characterization of deterministic sampling patterns for low-rank matrix completion , 2015, 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[14]  Bamdev Mishra,et al.  Fixed-rank matrix factorizations and Riemannian low-rank optimization , 2012, Comput. Stat..

[15]  P. Absil,et al.  Low-rank matrix completion via preconditioned optimization on the Grassmann manifold , 2015, Linear Algebra and its Applications.

[16]  Tengyu Ma,et al.  Matrix Completion has No Spurious Local Minimum , 2016, NIPS.

[17]  Emmanuel J. Candès,et al.  The Power of Convex Relaxation: Near-Optimal Matrix Completion , 2009, IEEE Transactions on Information Theory.

[18]  Benjamin Recht,et al.  A Simpler Approach to Matrix Completion , 2009, J. Mach. Learn. Res..

[19]  Yousef Saad,et al.  Scaled Gradients on Grassmann Manifolds for Matrix Completion , 2012, NIPS.

[20]  David Gross,et al.  Recovering Low-Rank Matrices From Few Coefficients in Any Basis , 2009, IEEE Transactions on Information Theory.

[21]  Christopher Ré,et al.  Parallel stochastic gradient algorithms for large-scale matrix completion , 2013, Mathematical Programming Computation.

[22]  Justin K. Romberg,et al.  An Overview of Low-Rank Matrix Recovery From Incomplete Observations , 2016, IEEE Journal of Selected Topics in Signal Processing.

[23]  J. Tanner,et al.  Low rank matrix completion by alternating steepest descent methods , 2016 .

[24]  Massimo Fornasier,et al.  Low-rank Matrix Recovery via Iteratively Reweighted Least Squares Minimization , 2010, SIAM J. Optim..

[25]  Michael A. Saunders,et al.  LSQR: An Algorithm for Sparse Linear Equations and Sparse Least Squares , 1982, TOMS.

[26]  Yin Zhang,et al.  Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm , 2012, Mathematical Programming Computation.

[27]  Christian Kümmerle,et al.  Harmonic Mean Iteratively Reweighted Least Squares for low-rank matrix recovery , 2017, 2017 International Conference on Sampling Theory and Applications (SampTA).

[28]  Jieping Ye,et al.  An accelerated gradient method for trace norm minimization , 2009, ICML '09.

[29]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[30]  Vikas Sindhwani,et al.  Efficient and Practical Stochastic Subgradient Descent for Nuclear Norm Regularization , 2012, ICML.

[31]  Takayuki Okatani,et al.  On the Wiberg Algorithm for Matrix Factorization in the Presence of Missing Components , 2007, International Journal of Computer Vision.

[32]  Jared Tanner,et al.  Normalized Iterative Hard Thresholding for Matrix Completion , 2013, SIAM J. Sci. Comput..

[33]  Moritz Hardt,et al.  Understanding Alternating Minimization for Matrix Completion , 2013, 2014 IEEE 55th Annual Symposium on Foundations of Computer Science.

[34]  Justin P. Haldar,et al.  Rank-Constrained Solutions to Linear Matrix Equations Using PowerFactorization , 2009, IEEE Signal Processing Letters.

[35]  Takayuki Okatani,et al.  Efficient algorithm for low-rank matrix factorization with missing components and performance comparison of latest algorithms , 2011, 2011 International Conference on Computer Vision.

[36]  Sujay Sanghavi,et al.  Completing any low-rank matrix, provably , 2013, J. Mach. Learn. Res..

[37]  Andrew W. Fitzgibbon,et al.  Damped Newton algorithms for matrix factorization with missing data , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[38]  Tony F. Chan,et al.  Guarantees of Riemannian Optimization for Low Rank Matrix Recovery , 2015, SIAM J. Matrix Anal. Appl..

[39]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2009, Found. Comput. Math..

[40]  Emmanuel J. Candès,et al.  Matrix Completion With Noise , 2009, Proceedings of the IEEE.

[41]  Bart Vandereycken,et al.  Low-Rank Matrix Completion by Riemannian Optimization , 2013, SIAM J. Optim..

[42]  Yuxin Chen,et al.  Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview , 2018, IEEE Transactions on Signal Processing.

[43]  Bamdev Mishra,et al.  R3MC: A Riemannian three-factor algorithm for low-rank matrix completion , 2013, 53rd IEEE Conference on Decision and Control.

[44]  Jeffrey D. Blanchard,et al.  CGIHT: Conjugate Gradient Iterative Hard Thresholding for Compressed Sensing and Matrix Completion , 2015 .

[45]  Shiqian Ma,et al.  Fixed point and Bregman iterative methods for matrix rank minimization , 2009, Math. Program..

[46]  Prateek Jain,et al.  Low-rank matrix completion using alternating minimization , 2012, STOC '13.

[47]  S. Yun,et al.  An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems , 2009 .

[48]  S. Szarek,et al.  Chapter 8 - Local Operator Theory, Random Matrices and Banach Spaces , 2001 .

[49]  Amit Singer,et al.  Uniqueness of Low-Rank Matrix Completion by Rigidity Theory , 2009, SIAM J. Matrix Anal. Appl..

[50]  Yuling Yan,et al.  Inference and uncertainty quantification for noisy matrix completion , 2019, Proceedings of the National Academy of Sciences.