Fast and Sample-Efficient Federated Low Rank Matrix Recovery From Column-Wise Linear and Quadratic Projections

We study the following lesser-known low rank (LR) recovery problem: recover an <inline-formula> <tex-math notation="LaTeX">$n \times q$ </tex-math></inline-formula> rank-<inline-formula> <tex-math notation="LaTeX">$r$ </tex-math></inline-formula> matrix, <inline-formula> <tex-math notation="LaTeX">${ \boldsymbol {X}}^{\ast}=[\boldsymbol {x}^{\ast}_{1}, \boldsymbol {x}^{\ast}_{2}, \ldots, \boldsymbol {x}^{\ast}_{q}]$ </tex-math></inline-formula>, with <inline-formula> <tex-math notation="LaTeX">$r \ll \min (n,q)$ </tex-math></inline-formula>, from <inline-formula> <tex-math notation="LaTeX">$m$ </tex-math></inline-formula> independent linear projections of each of its <inline-formula> <tex-math notation="LaTeX">$q$ </tex-math></inline-formula> columns, i.e., from <inline-formula> <tex-math notation="LaTeX">$\boldsymbol {y}_{k}:= \boldsymbol {A}_{k} \boldsymbol {x}^{\ast}_{k}, k \in [q]$ </tex-math></inline-formula>, when <inline-formula> <tex-math notation="LaTeX">$\boldsymbol {y}_{k}$ </tex-math></inline-formula> is an <inline-formula> <tex-math notation="LaTeX">$m$ </tex-math></inline-formula>-length vector with <inline-formula> <tex-math notation="LaTeX">$m < n$ </tex-math></inline-formula>. The matrices <inline-formula> <tex-math notation="LaTeX">$\boldsymbol {A}_{k}$ </tex-math></inline-formula> are known and mutually independent for different <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>. We introduce a novel gradient descent (GD) based solution called AltGD-Min. We show that, if the <inline-formula> <tex-math notation="LaTeX">$\boldsymbol {A}_{k}\text{s}$ </tex-math></inline-formula> are i.i.d. with i.i.d. Gaussian entries, and if the right singular vectors of <inline-formula> <tex-math notation="LaTeX">${ \boldsymbol {X}}^{\ast}$ </tex-math></inline-formula> satisfy the incoherence assumption, then <inline-formula> <tex-math notation="LaTeX">$\epsilon $ </tex-math></inline-formula>-accurate recovery of <inline-formula> <tex-math notation="LaTeX">${ \boldsymbol {X}}^{\ast}$ </tex-math></inline-formula> is possible with order <inline-formula> <tex-math notation="LaTeX">$(n+q) r^{2} \log (1/\epsilon)$ </tex-math></inline-formula> total samples and order <inline-formula> <tex-math notation="LaTeX">$mq nr \log (1/\epsilon)$ </tex-math></inline-formula> time. Compared with existing work, this is the fastest solution. For <inline-formula> <tex-math notation="LaTeX">$\epsilon < r^{1/4}$ </tex-math></inline-formula>, it also has the best sample complexity. A simple extension of AltGD-Min also provably solves LR Phase Retrieval, which is a magnitude-only generalization of the above problem. AltGD-Min factorizes the unknown <inline-formula> <tex-math notation="LaTeX">${ \boldsymbol {X}}$ </tex-math></inline-formula> as <inline-formula> <tex-math notation="LaTeX">${ \boldsymbol {X}}= { \boldsymbol {U}} \boldsymbol {B} $ </tex-math></inline-formula> where <inline-formula> <tex-math notation="LaTeX">${ \boldsymbol {U}}$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$\boldsymbol {B}$ </tex-math></inline-formula> are matrices with <inline-formula> <tex-math notation="LaTeX">$r$ </tex-math></inline-formula> columns and rows respectively. It alternates between a (projected) GD step for updating <inline-formula> <tex-math notation="LaTeX">${ \boldsymbol {U}}$ </tex-math></inline-formula>, and a minimization step for updating <inline-formula> <tex-math notation="LaTeX">$\boldsymbol {B}$ </tex-math></inline-formula>. Its each iteration is as fast as that of regular projected GD because the minimization over <inline-formula> <tex-math notation="LaTeX">$\boldsymbol {B}$ </tex-math></inline-formula> decouples column-wise. At the same time, we can prove exponential error decay for it, which we are unable to for projected GD. Finally, it can also be efficiently federated with a communication cost of only <inline-formula> <tex-math notation="LaTeX">$nr$ </tex-math></inline-formula> per node, instead of <inline-formula> <tex-math notation="LaTeX">$nq$ </tex-math></inline-formula> for projected GD.

[1]  N. Vaswani,et al.  Fast Low Rank Column-Wise Compressive Sensing for Accelerated Dynamic MRI , 2022, IEEE Transactions on Computational Imaging.

[2]  Seyedehsara Nayer,et al.  Sample-Efficient Low Rank Phase Retrieval , 2021, International Symposium on Information Theory.

[3]  Jianqing Fan,et al.  Spectral Methods for Data Science: A Statistical Perspective , 2020, Found. Trends Mach. Learn..

[4]  O. Papaspiliopoulos.  High-Dimensional Probability: An Introduction with Applications in Data Science , 2020 .

[5]  Seyedehsara Nayer,et al.  Sample-Efficient Low Rank Phase Retrieval , 2020, IEEE Transactions on Information Theory.

[6]  A. Ramamoorthy,et al.  Federated Over-the-Air Subspace Learning from Incomplete Data , 2020, ArXiv.

[7]  Jon Crowcroft,et al.  Federated Principal Component Analysis , 2019, NeurIPS.

[8]  Cecilia Mascolo,et al.  Federated PCA with Adaptive Rank Estimation , 2019, ArXiv.

[9]  Seyedehsara Nayer,et al.  Provable Low Rank Phase Retrieval , 2019, IEEE Transactions on Information Theory.

[10]  Junzhou Huang,et al.  An efficient algorithm for dynamic MRI using low‐rank and total variation regularizations , 2018, Medical Image Anal..

[11]  Yuxin Chen,et al.  Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution , 2017, Found. Comput. Math..

[12]  Yonina C. Eldar,et al.  Phaseless PCA: Low-Rank Matrix Recovery from Column-wise Phaseless Measurements , 2019, ICML.

[13]  Prateek Jain,et al.  Nearly Optimal Robust Matrix Completion , 2016, ICML.

[14]  Constantine Caramanis,et al.  Fast Algorithms for Robust PCA via Gradient Descent , 2016, NIPS.

[15]  John D. Lafferty,et al.  Convergence Analysis for Rectangular Matrix Completion Using Burer-Monteiro Factorization and Gradient Descent , 2016, ArXiv.

[16]  Xiaodong Li,et al.  Optimal Rates of Convergence for Noisy Sparse Phase Retrieval via Thresholded Wirtinger Flow , 2015, ArXiv.

[17]  Prateek Jain,et al.  Fast Exact Matrix Completion with Finite Samples , 2014, COLT.

[18]  Shannon M. Hughes,et al.  Memory and Computation Efficient PCA via Very Sparse Random Projections , 2014, ICML.

[19]  Mary Wootters,et al.  Fast matrix completion without the condition number , 2014, COLT.

[20]  Akshay Krishnamurthy,et al.  Subspace learning from extremely compressed measurements , 2014, 2014 48th Asilomar Conference on Signals, Systems and Computers.

[21]  Moritz Hardt,et al.  The Noisy Power Method: A Meta Algorithm with Applications , 2013, NIPS.

[22]  Andrea J. Goldsmith,et al.  Exact and Stable Covariance Estimation From Quadratic Sampling via Convex Programming , 2013, IEEE Transactions on Information Theory.

[23]  Prateek Jain,et al.  Phase Retrieval Using Alternating Minimization , 2013, IEEE Transactions on Signal Processing.

[24]  Prateek Jain,et al.  Low-rank matrix completion using alternating minimization , 2012, STOC '13.

[25]  Michael B. Wakin,et al.  Sketched SVD: Recovering Spectral Features from Compressive Measurements , 2012, ArXiv.

[26]  Hanchao Qi,et al.  Invariance of principal components under low-dimensional random projection of the data , 2012, 2012 19th IEEE International Conference on Image Processing.

[27]  H. Yau,et al.  Spectral statistics of Erdős–Rényi graphs I: Local semicircle law , 2011, 1103.1919.

[28]  Mathews Jacob,et al.  Accelerated Dynamic MRI Exploiting Sparsity and Low-Rank Structure: k-t SLR , 2011, IEEE Transactions on Medical Imaging.

[29]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.

[30]  Joel A. Tropp,et al.  User-Friendly Tail Bounds for Sums of Random Matrices , 2010, Found. Comput. Math..

[31]  Martin J. Wainwright,et al.  Estimation of (near) low-rank matrices with noise and high-dimensional scaling , 2009, ICML.

[32]  Kay Nehrke,et al.  k‐t PCA: Temporally constrained k‐t BLAST reconstruction using principal component analysis , 2009, Magnetic resonance in medicine.

[33]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2008, Found. Comput. Math..

[34]  D. Donoho,et al.  Sparse MRI: The application of compressed sensing for rapid MR imaging , 2007, Magnetic resonance in medicine.

[35]  Zhi-Pei Liang,et al.  Spatiotemporal Imaging with Partially Separable Functions , 2007, 2007 Joint Meeting of the 6th International Symposium on Noninvasive Functional Source Imaging of the Brain and Heart and the International Conference on Functional Biomedical Imaging.

[36]  A. Mukherjea,et al.  Real and Functional Analysis , 1978 .

[37]  P. Wedin.  Perturbation bounds in connection with singular value decomposition , 1972 .

[38]  Namrata Vaswani,et al.  Sample Efficient Fourier Ptychography for Structured Data , 2020, IEEE Transactions on Computational Imaging.

[39]  W. Hager,et al.  and s , 2019, Shallow Water Hydraulics.

[40]  Justin Romberg,et al.  Decentralized sketching of low rank matrices , 2019, NeurIPS.

[41]  Yingbin Liang,et al.  A Nonconvex Approach for Phase Retrieval: Reshaped Wirtinger Flow and Incremental Algorithms , 2017, J. Mach. Learn. Res..

[42]  W. Marsden.  I and J , 2012 .

[43]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.