A statistical physics approach to learning curves for the inverse Ising problem

Using methods of statistical physics, we analyse the error of learning couplings in large Ising models from independent data (the inverse Ising problem). We concentrate on learning based on local cost functions, such as the pseudo-likelihood method for which the couplings are inferred independently for each spin. Assuming that the data are generated from a true Ising model, we compute the reconstruction error of the couplings using a combination of the replica method with the cavity approach for densely connected systems. We show that an explicit estimator based on a quadratic cost function achieves minimal reconstruction error, but requires the length of the true coupling vector as prior knowledge. A simple mean field estimator of the couplings which does not need such knowledge is asymptotically optimal, i.e. when the number of observations is much large than the number of spins. Comparison of the theory with numerical simulations shows excellent agreement for data generated from two models with random couplings in the high temperature region: a model with independent couplings (Sherrington-Kirkpatrick model), and a model where the matrix of couplings has a Wishart distribution.

[1]  Andrea Pagnani,et al.  Regularization and decimation pseudolikelihood approaches to statistical inference inXYspin models , 2016, 1603.05101.

[2]  Nestor Caticha,et al.  LEARNING A SPIN GLASS : DETERMINING HAMILTONIANS FROM METASTABLE STATES , 1998 .

[3]  Manfred Opper,et al.  Statistical mechanics of generalization , 1998 .

[4]  Federico Ricci-Tersenghi,et al.  Pseudolikelihood decimation algorithm improving the inference of the interaction network in a general class of Ising models. , 2013, Physical review letters.

[5]  R. Monasson,et al.  Small-correlation expansions for the inverse Ising problem , 2008, 0811.3574.

[6]  H. Nishimori.  Statistical Physics of Spin Glasses and Information Processing , 2001 .

[7]  M. Mézard,et al.  Spin Glass Theory And Beyond: An Introduction To The Replica Method And Its Applications , 1986 .

[8]  Toshiyuki TANAKA.  Mean-field theory of Boltzmann machine learning , 1998 .

[9]  M. Opper,et al.  Adaptive and self-averaging Thouless-Anderson-Palmer mean-field theory for probabilistic modeling. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[10]  Sompolinsky,et al.  Statistical mechanics of learning from examples. , 1992, Physical review. A, Atomic, molecular, and optical physics.

[11]  Manfred Opper,et al.  Learning of couplings for random asymmetric kinetic Ising models revisited: random correlation matrices and learning curves , 2015, 1508.05865.

[12]  Simona Cocco,et al.  Adaptive Cluster Expansion for Inferring Boltzmann Machines with Noisy Data , 2011, Physical review letters.

[13]  Jascha Sohl-Dickstein,et al.  A new method for parameter estimation in probabilistic models: Minimum probability flow , 2011, Physical review letters.

[14]  G. Parisi,et al.  Mean-field equations for spin models with orthogonal interaction matrices , 1995, cond-mat/9503009.

[15]  A. Bray,et al.  Metastable states in spin glasses , 1980 .

[16]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[17]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[18]  J. Baik,et al.  The Oxford Handbook of Random Matrix Theory , 2011 .

[19]  Christian Van den Broeck,et al.  Statistical Mechanics of Learning , 2001 .

[20]  Karim M. Abadir,et al.  Matrix Algebra: Notation , 2005 .

[21]  J. Besag.  Spatial Interaction and the Statistical Analysis of Lattice Systems , 1974 .

[22]  Erik Aurell,et al.  Frontiers in Computational Neuroscience , 2022 .

[23]  F. Ricci-Tersenghi.  The Bethe approximation for solving the inverse Ising problem: a comparison with other inference methods , 2011, 1112.4814.

[24]  O. Kinouchi,et al.  Optimal generalization in perceptions , 1992 .

[25]  S. Kirkpatrick,et al.  Solvable Model of a Spin-Glass , 1975 .

[26]  M. Mézard,et al.  The Bethe lattice spin glass revisited , 2000, cond-mat/0009418.

[27]  B. H.J.KappentandF..  Efficient learning in Boltzmann Machines using linear response theory* , 2018 .

[28]  M. Schervish.  Theory of Statistics , 1995 .

[29]  J. Hertz,et al.  Ising model for neural data: model quality and approximate methods for extracting functional connectivity. , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[30]  E. Aurell,et al.  Inverse Ising inference using all the data. , 2011, Physical review letters.

[31]  Michael J. Berry,et al.  Weak pairwise correlations imply strongly correlated network states in a neural population , 2005, Nature.

[32]  A. Maritan,et al.  Using the principle of entropy maximization to infer genetic interaction networks from gene expression patterns , 2006, Proceedings of the National Academy of Sciences.

[33]  Onur Dikmen,et al.  Consistent inference of a general model using the pseudolikelihood method. , 2014, Physical review. E, Statistical, nonlinear, and soft matter physics.

[34]  Surya Ganguli,et al.  Statistical Mechanics of Optimal Convex Inference in High Dimensions , 2016 .

[35]  Robert E. Schapire,et al.  Faster solutions of the inverse pairwise Ising problem , 2008 .

[36]  J. Berg,et al.  Bethe–Peierls approximation and the inverse Ising problem , 2011, 1112.3501.

[37]  Sompolinsky,et al.  Spin-glass models of neural networks. , 1985, Physical review. A, General physics.

[38]  I. Kanter.  Book Review: Statistical Mechanics of Learning. By A. Engel and C. Van den Broeck, Cambridge University Press , 2001 .

[39]  T. Hwa,et al.  Identification of direct residue contacts in protein–protein interaction by message passing , 2009, Proceedings of the National Academy of Sciences.

[40]  Ole Winther,et al.  A theory of solving TAP equations for Ising models with general invariant random matrices , 2015, ArXiv.

[41]  M. Opper,et al.  5 Statistical Mechanics of Generalization , .

[42]  J. Berg,et al.  Statistical mechanics of the inverse Ising problem and the optimal objective function , 2016, 1611.04281.

[43]  P. Deift,et al.  Random Matrix Theory: Invariant Ensembles and Universality , 2009 .