Adaptive and self-averaging Thouless-Anderson-Palmer mean-field theory for probabilistic modeling.

We develop a generalization of the Thouless-Anderson-Palmer (TAP) mean-field approach of disorder physics, which makes the method applicable to the computation of approximate averages in probabilistic models for real data. In contrast to the conventional TAP approach, where the knowledge of the distribution of couplings between the random variables is required, our method adapts to the concrete set of couplings. We show the significance of the approach in two ways: Our approach reproduces replica symmetric results for a wide class of toy models (assuming a nonglassy phase) with given disorder distributions in the thermodynamic limit. On the other hand, simulations on a real data model demonstrate that the method achieves more accurate predictions as compared to conventional TAP approaches.

[1]  S. Kirkpatrick,et al.  Solvable Model of a Spin-Glass , 1975 .

[2]  R. Palmer,et al.  Solution of 'Solvable model of a spin glass' , 1977 .

[3]  C. Dominicis.  Toward a mean field theory of spin glasses: The TAP route revisited , 1980 .

[4]  A. Bray,et al.  Metastable states in spin glasses , 1980 .

[5]  T. Plefka.  Convergence condition of the TAP equation for the infinite-ranged Ising spin glass model , 1982 .

[6]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[7]  M. Mézard,et al.  Replicas and optimization , 1985 .

[8]  M. Mézard,et al.  Spin Glass Theory and Beyond , 1987 .

[9]  Terrence J. Sejnowski,et al.  Analysis of hidden units in a layered network trained to classify sonar targets , 1988, Neural Networks.

[10]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[11]  M. Mézard.  The space of interactions in neural networks: Gardner's computation with the cavity method , 1989 .

[12]  Sompolinsky,et al.  Statistical mechanics of learning from examples. , 1992, Physical review. A, Atomic, molecular, and optical physics.

[13]  Radford M. Neal.  Connectionist Learning of Belief Networks , 1992, Artif. Intell..

[14]  Carsten Peterson,et al.  Neural Networks for Optimization Problems with Inequality Constraints: The Knapsack Problem , 1993, Neural Computation.

[15]  G. Parisi,et al.  Replica field theory for deterministic models: II. A non-random spin glass with glassy behaviour , 1994, cond-mat/9406074.

[16]  K. Y. Wong.  Microscopic Equations and Stability Conditions in Optimal Neural Networks , 1995 .

[17]  G. Parisi,et al.  Mean-field equations for spin models with orthogonal interaction matrices , 1995, cond-mat/9503009.

[18]  Michael I. Jordan,et al.  Mean Field Theory for Sigmoid Belief Networks , 1996, J. Artif. Intell. Res..

[19]  Opper.  On-line versus Off-line Learning from Random Examples: General Results. , 1996, Physical review letters.

[20]  Opper,et al.  Mean field approach to Bayes learning in feed-forward neural networks. , 1996, Physical review letters.

[21]  Yoshiyuki Kabashima,et al.  Belief propagation vs. TAP for decoding corrupted messages , 1998 .

[22]  Hilbert J. Kappen,et al.  Efficient Learning in Boltzmann Machines Using Linear Response Theory , 1998, Neural Computation.

[23]  Toshiyuki TANAKA.  Mean-field theory of Boltzmann machine learning , 1998 .

[24]  Te-Won Lee.  Independent Component Analysis , 1998, Springer US.

[25]  S. Keerthi,et al.  Information geometry and Plefka's mean-field theory , 2000 .

[26]  Ole Winther,et al.  Gaussian Processes for Classification: Mean-Field Algorithms , 2000, Neural Computation.

[27]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[28]  M. Opper,et al.  Tractable approximations for probabilistic models: the adaptive Thouless-Anderson-Palmer mean field approach. , 2001, Physical review letters.

[29]  William H. Press,et al.  Numerical recipes in C , 2002 .