Fundamental limits of low-rank matrix estimation

We consider the high-dimensional inference problem where the signal is a low-rank matrix which is corrupted by an additive Gaussian noise. Given a probabilistic model for the low-rank matrix, we compute the limit in the large dimension setting for the mutual information between the signal and the observations, as well as the matrix minimum mean square error, while the rank of the signal remains constant. This allows to locate the information-theoretic threshold for this estimation problem, i.e. the critical value of the signal intensity below which it is impossible to recover the low-rank matrix.

[1]  S. Edwards,et al.  The eigenvalue spectrum of a large symmetric random matrix , 1976 .

[2]  R. Palmer,et al.  Solution of 'Solvable model of a spin glass' , 1977 .

[3]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[4]  M. Mézard,et al.  Spin Glass Theory And Beyond: An Introduction To The Replica Method And Its Applications , 1986 .

[5]  F. Guerra,et al.  General properties of overlap probability distributions in disordered spin systems. Towards Parisi ultrametricity , 1998, cond-mat/9807333.

[6]  Y. Iba.  The Nishimori line and Bayesian statistics , 1998, cond-mat/9809190.

[7]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[8]  H. Nishimori.  Statistical Physics of Spin Glasses and Information Processing , 2001 .

[9]  I. Johnstone.  On the distribution of the largest eigenvalue in principal components analysis , 2001 .

[10]  西森 秀稔.  Statistical physics of spin glasses and information processing : an introduction , 2001 .

[11]  Paul R. Milgrom,et al.  Envelope Theorems for Arbitrary Choice Sets , 2002 .

[12]  F. Guerra.  Broken Replica Symmetry Bounds in the Mean Field Spin Glass Model , 2002, cond-mat/0205123.

[13]  M. Aizenman,et al.  Extended variational principle for the Sherrington-Kirkpatrick spin-glass model , 2003 .

[14]  S. Péché,et al.  Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices , 2004, math/0403022.

[15]  Shlomo Shamai,et al.  Mutual information and minimum mean-square error in Gaussian channels , 2004, IEEE Transactions on Information Theory.

[16]  D. Féral,et al.  The Largest Eigenvalue of Rank One Deformation of Large Wigner Matrices , 2006, math/0605624.

[17]  Andrea Montanari,et al.  Estimating random variables from random sparse observations , 2007, Eur. Trans. Telecommun..

[18]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[19]  Satish Babu Korada,et al.  Exact Solution of the Gauge Symmetric p-Spin Glass Model on a Complete Graph , 2009 .

[20]  A. Barra,et al.  The Replica Symmetric Approximation of the Analogical Neural Network , 2009, 0911.3096.

[21]  Andrea Montanari,et al.  Message passing algorithms for compressed sensing: I. motivation and construction , 2009, 2010 IEEE Information Theory Workshop on Information Theory (ITW 2010, Cairo).

[22]  Nicolas Macris,et al.  Tight Bounds on the Capacity of Binary Input Random CDMA Systems , 2008, IEEE Transactions on Information Theory.

[23]  A. Barra,et al.  Equilibrium statistical mechanics of bipartite spin systems , 2010, 1012.1261.

[24]  Adel Javanmard,et al.  State Evolution for General Approximate Message Passing Algorithms, with Applications to Spatial Coupling , 2012, ArXiv.

[25]  Sundeep Rangan,et al.  Iterative estimation of constrained rank-one matrices in noise , 2012, 2012 IEEE International Symposium on Information Theory Proceedings.

[26]  Raj Rao Nadakuditi,et al.  The singular values and vectors of low rank perturbations of large rectangular random matrices , 2011, J. Multivar. Anal..

[27]  D. Panchenko.  The Sherrington-Kirkpatrick Model , 2013 .

[28]  Toshiyuki Tanaka,et al.  Low-rank matrix reconstruction and clustering via approximate message passing , 2013, NIPS.

[29]  Gábor Lugosi,et al.  Concentration Inequalities - A Nonasymptotic Theory of Independence , 2013, Concentration Inequalities.

[30]  Andrea Montanari,et al.  Information-theoretically optimal sparse PCA , 2014, 2014 IEEE International Symposium on Information Theory.

[31]  Florent Krzakala,et al.  Phase transitions in sparse PCA , 2015, 2015 IEEE International Symposium on Information Theory (ISIT).

[32]  Florent Krzakala,et al.  Statistical physics of inference: thresholds and algorithms , 2015, ArXiv.

[33]  Florent Krzakala,et al.  MMSE of probabilistic low-rank matrix estimation: Universality with respect to the output channel , 2015, 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[34]  Florent Krzakala,et al.  Mutual information in rank-one matrix estimation , 2016, 2016 IEEE Information Theory Workshop (ITW).

[35]  Jess Banks,et al.  Phase transitions and optimal algorithms in high-dimensional Gaussian mixture clustering , 2016, 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[36]  Afonso S. Bandeira,et al.  Statistical limits of spiked tensor models , 2016, Annales de l'Institut Henri Poincaré, Probabilités et Statistiques.

[37]  Ankur Moitra,et al.  Optimality and Sub-optimality of PCA for Spiked Random Matrices and Synchronization , 2016, ArXiv.

[38]  Nicolas Macris,et al.  Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula , 2016, NIPS.

[39]  Marc Lelarge,et al.  Fundamental limits of symmetric low-rank matrix estimation , 2016, Probability Theory and Related Fields.

[40]  Florent Krzakala,et al.  Constrained low-rank matrix estimation: phase transitions, approximate message passing and applications , 2017, ArXiv.

[41]  Jess Banks,et al.  Information-theoretic bounds and phase transitions in clustering, sparse PCA, and submatrix localization , 2017, ISIT.

[42]  F. Krzakala,et al.  Information-theoretic thresholds from the cavity method , 2016, Symposium on the Theory of Computing.