Notes on computational-to-statistical gaps: predictions using statistical physics

In these notes we describe heuristics to predict computational-to-statistical gaps in certain statistical problems. These are regimes in which the underlying statistical problem is information-theoretically possible although no efficient algorithm exists, rendering the problem essentially unsolvable for large instances. The methods we describe here are based on mature, albeit non-rigorous, tools from statistical physics. These notes are based on a lecture series given by the authors at the Courant Institute of Mathematical Sciences in New York City, on May 16th, 2017.

[1]  Andrea Montanari,et al.  Asymptotic mutual information for the binary stochastic block model , 2016, 2016 IEEE International Symposium on Information Theory (ISIT).

[2]  David Steurer,et al.  Dictionary Learning and Tensor Decomposition via the Sum-of-Squares Method , 2014, STOC.

[3]  Florent Krzakala,et al.  Phase transitions in sparse PCA , 2015, 2015 IEEE International Symposium on Information Theory (ISIT).

[4]  Florent Krzakala,et al.  Statistical and computational phase transitions in spiked tensor estimation , 2017, 2017 IEEE International Symposium on Information Theory (ISIT).

[5]  Adel Javanmard,et al.  Phase transitions in semidefinite relaxations , 2015, Proceedings of the National Academy of Sciences.

[6]  Samuel B. Hopkins,et al.  Bayesian estimation from few samples: community detection and related problems , 2017, ArXiv.

[7]  Emmanuel Abbe,et al.  Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic BP, and the information-computation gap , 2015, ArXiv.

[8]  David P. Williamson,et al.  Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming , 1995, JACM.

[9]  Jean B. Lasserre,et al.  Global Optimization with Polynomials and the Problem of Moments , 2000, SIAM J. Optim..

[10]  Yi Zheng,et al.  No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis , 2017, ICML.

[11]  Cristopher Moore,et al.  The Computer Science and Physics of Community Detection: Landscapes, Phase Transitions, and Hardness , 2017, Bull. EATCS.

[12]  Florent Krzakala,et al.  Statistical physics of inference: thresholds and algorithms , 2015, ArXiv.

[13]  Cristopher Moore,et al.  Phase transition in the detection of modules in sparse networks , 2011, Physical review letters.

[14]  Nicolas Boumal,et al.  On the low-rank approach for semidefinite programs arising in synchronization and community detection , 2016, COLT.

[15]  Tengyu Ma,et al.  Polynomial-Time Tensor Decompositions with Sum-of-Squares , 2016, 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS).

[16]  Florent Krzakala,et al.  MMSE of probabilistic low-rank matrix estimation: Universality with respect to the output channel , 2015, 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[17]  Nicolas Macris,et al.  Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula , 2016, NIPS.

[18]  H. Nishimori.  Exact results and critical properties of the Ising model with competing interactions , 1980 .

[19]  Andrea Montanari,et al.  Information-theoretically optimal sparse PCA , 2014, 2014 IEEE International Symposium on Information Theory.

[20]  Joan Bruna,et al.  Neural Networks with Finite Intrinsic Dimension have no Spurious Valleys , 2018, ArXiv.

[21]  Subhash Khot.  On the power of unique 2-prover 1-round games , 2002, STOC '02.

[22]  Jonathan Shi,et al.  Tensor principal component analysis via sum-of-square proofs , 2015, COLT.

[23]  西森 秀稔.  Statistical physics of spin glasses and information processing : an introduction , 2001 .

[24]  Giorgio Parisi,et al.  Infinite Number of Order Parameters for Spin-Glasses , 1979 .

[25]  Yurii Nesterov,et al.  Squared Functional Systems and Optimization Problems , 2000 .

[26]  S. Stenholm.  Information, Physics and Computation, by Marc Mézard and Andrea Montanari , 2010 .

[27]  P. Rigollet,et al.  Optimal detection of sparse principal components in high dimension , 2012, 1202.5070.

[28]  Yihong Wu,et al.  Computational Barriers in Minimax Submatrix Detection , 2013, ArXiv.

[29]  Philippe Rigollet,et al.  Complexity Theoretic Lower Bounds for Sparse Principal Component Detection , 2013, COLT.

[30]  Pravesh Kothari,et al.  A Nearly Tight Sum-of-Squares Lower Bound for the Planted Clique Problem , 2016, 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS).

[31]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[32]  H. Nishimori.  Internal Energy, Specific Heat and Correlation Function of the Bond-Random Ising Model , 1981 .

[33]  Tengyu Ma,et al.  Decomposing Overcomplete 3rd Order Tensors using Sum-of-Squares Algorithms , 2015, APPROX-RANDOM.

[34]  Bruce E. Hajek,et al.  Computational Lower Bounds for Community Detection on Random Graphs , 2014, COLT.

[35]  Tengyu Ma,et al.  On the optimization landscape of tensor decompositions , 2017, Mathematical Programming.

[36]  Adel Javanmard,et al.  State Evolution for General Approximate Message Passing Algorithms, with Applications to Spatial Coupling , 2012, ArXiv.

[37]  László Lovász,et al.  On the Shannon capacity of a graph , 1979, IEEE Trans. Inf. Theory.

[38]  Judea Pearl,et al.  Fusion, Propagation, and Structuring in Belief Networks , 1986, Artif. Intell..

[39]  Emmanuel Abbe,et al.  Community detection and stochastic block models: recent developments , 2017, Found. Trends Commun. Inf. Theory.

[40]  Nicolas Boumal,et al.  The non-convex Burer-Monteiro approach works on smooth semidefinite programs , 2016, NIPS.

[41]  Subhash Khot.  On the Unique Games Conjecture (Invited Survey) , 2010, Computational Complexity Conference.

[42]  Andrea Montanari,et al.  The dynamics of message passing on dense graphs, with applications to compressed sensing , 2010, ISIT.

[43]  R. Palmer,et al.  Solution of 'Solvable model of a spin glass' , 1977 .

[44]  Jess Banks,et al.  Information-theoretic thresholds for community detection in sparse networks , 2016, COLT.

[45]  Elchanan Mossel,et al.  A Proof of the Block Model Threshold Conjecture , 2013, Combinatorica.

[46]  David Steurer,et al.  Sum-of-squares proofs and the quest toward optimal algorithms , 2014, Electron. Colloquium Comput. Complex..

[47]  David Steurer,et al.  Rounding sum-of-squares relaxations , 2013, Electron. Colloquium Comput. Complex..

[48]  N. Z. Shor.  An approach to obtaining global extremums in polynomial mathematical programming problems , 1987 .

[49]  E. Bolthausen.  An Iterative Construction of Solutions of the TAP Equations for the Sherrington–Kirkpatrick Model , 2012, 1201.2891.

[50]  P. Parrilo.  Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization , 2000 .

[51]  H. Kesten,et al.  Additional Limit Theorems for Indecomposable Multidimensional Galton-Watson Processes , 1966 .

[52]  Tselil Schramm,et al.  Fast spectral algorithms from sum-of-squares proofs: tensor decomposition and planted sparse vectors , 2015, STOC.

[53]  R. Adler,et al.  Random Fields and Geometry , 2007 .

[54]  Ankur Moitra,et al.  Optimality and Sub-optimality of PCA for Spiked Random Matrices and Synchronization , 2016, ArXiv.

[55]  Afonso S. Bandeira,et al.  Statistical limits of spiked tensor models , 2016, Annales de l'Institut Henri Poincaré, Probabilités et Statistiques.

[56]  Antonio Auffinger,et al.  Random Matrices and Complexity of Spin Glasses , 2010, 1003.1129.

[57]  H. Nishimori.  Statistical Physics of Spin Glasses and Information Processing , 2001 .

[58]  Kathryn B. Laskey,et al.  Stochastic blockmodels: First steps , 1983 .

[59]  Giorgio Parisi,et al.  SK Model: The Replica Solution without Replicas , 1986 .

[60]  Michel X. Goemans,et al.  Community detection in hypergraphs, spiked tensor models, and Sum-of-Squares , 2017, 2017 International Conference on Sampling Theory and Applications (SampTA).

[61]  A. Singer.  Angular Synchronization by Eigenvectors and Semidefinite Programming. , 2009, Applied and computational harmonic analysis.

[62]  Andrea Montanari,et al.  Non-Negative Principal Component Analysis: Message Passing Algorithms and Sharp Asymptotics , 2014, IEEE Transactions on Information Theory.

[63]  Ankur Moitra,et al.  Noisy tensor completion via the sum-of-squares hierarchy , 2015, Mathematical Programming.

[64]  Sundeep Rangan,et al.  Iterative Reconstruction of Rank-One Matrices in Noise , 2012 .

[65]  David Steurer,et al.  Exact tensor completion with sum-of-squares , 2017, COLT.

[66]  Laurent Massoulié,et al.  Community detection thresholds and the weak Ramanujan property , 2013, STOC.

[67]  Ankur Moitra,et al.  Message‐Passing Algorithms for Synchronization Problems over Compact Groups , 2016, ArXiv.

[68]  Andrea Montanari,et al.  A statistical model for tensor PCA , 2014, NIPS.

[69]  Noga Alon,et al.  Finding a large hidden clique in a random graph , 1998, SODA '98.

[70]  Prasad Raghavendra,et al.  Optimal algorithms and inapproximability results for every CSP? , 2008, STOC.

[71]  Marc Lelarge,et al.  Fundamental limits of symmetric low-rank matrix estimation , 2016, Probability Theory and Related Fields.

[72]  Florent Krzakala,et al.  Mutual information in rank-one matrix estimation , 2016, 2016 IEEE Information Theory Workshop (ITW).