Approximate Learning Algorithm in Boltzmann Machines

Boltzmann machines can be regarded as Markov random fields. For binary cases, they are equivalent to the Ising spin model in statistical mechanics. Learning systems in Boltzmann machines are one of the NP-hard problems. Thus, in general we have to use approximate methods to construct practical learning algorithms in this context. In this letter, we propose new and practical learning algorithms for Boltzmann machines by using the belief propagation algorithm and the linear response approximation, which are often referred as advanced mean field methods. Finally, we show the validity of our algorithm using numerical experiments.

[1]  R. Palmer,et al.  Solution of 'Solvable model of a spin glass' , 1977 .

[2]  Alessandro Pelizzola,et al.  Cluster Variation Method in Statistical Physics and Probabilistic Graphical Models , 2005, ArXiv.

[3]  Yee Whye Teh,et al.  Approximate inference in Boltzmann machines , 2003, Artif. Intell..

[4]  Yee Whye Teh,et al.  Linear Response Algorithms for Approximate Inference in Graphical Models , 2004, Neural Computation.

[5]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[6]  Max Welling,et al.  Learning in Markov Random Fields An Empirical Study , 2005 .

[7]  Tsuyoshi Horiguchi.  On the bethe approximation for the random bond Ising model , 1981 .

[8]  Kazuyuki Tanaka,et al.  Probabilistic Inference by means of Cluster Variation Method and Linear Response Theory , 2003 .

[9]  T. Plefka.  Convergence condition of the TAP equation for the infinite-ranged Ising spin glass model , 1982 .

[10]  M. Opper,et al.  Advanced mean field methods: theory and practice , 2001 .

[11]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[12]  Martin J. Wainwright,et al.  Estimating the "Wrong" Graphical Model: Benefits in the Computation-Limited Setting , 2006, J. Mach. Learn. Res..

[13]  Toshiyuki TANAKA.  Mean-field theory of Boltzmann machine learning , 1998 .

[14]  Muneki Yasuda,et al.  The Relationship between Plefka's Expansion and the Cluster Variation Method , 2006 .

[15]  Martin J. Wainwright,et al.  Tree-reweighted belief propagation algorithms and approximate ML estimation by pseudo-moment matching , 2003, AISTATS.

[16]  Hilbert J. Kappen,et al.  Efficient Learning in Boltzmann Machines Using Linear Response Theory , 1998, Neural Computation.

[17]  Muneki Yasuda,et al.  The mathematical structure of the approximate linear response relation , 2007 .

[18]  Carsten Peterson,et al.  A Mean Field Theory Learning Algorithm for Neural Networks , 1987, Complex Syst..

[19]  Tsuyoshi Horiguchi,et al.  Triangular approximation for Ising model and its application to Boltzmann machine , 2006 .

[20]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[21]  S. Kirkpatrick,et al.  Solvable Model of a Spin-Glass , 1975 .

[22]  T. Morita,et al.  Exactly solvable model of a spin glass , 1976 .