Learning in higher order Boltzmann machines using linear response

We introduce an efficient method for learning and inference in higher order Boltzmann machines. The method is based on mean field theory with the linear response correction. We compute the correlations using the exact and the approximated method for a fully connected third order network of ten neurons. In addition, we compare the results of the exact and approximate learning algorithm. Finally we use the presented method to solve the shifter problem. We conclude that the linear response approximation gives good results as long as the couplings are not too large.

[1]  T. Plefka.  Convergence condition of the TAP equation for the infinite-ranged Ising spin glass model , 1982 .

[2]  R. Palmer,et al.  Solution of 'Solvable model of a spin glass' , 1977 .

[3]  L. Onsager.  Electric Moments of Molecules in Liquids , 1936 .

[4]  Shun-ichi Amari,et al.  Information geometry of Boltzmann machines , 1992, IEEE Trans. Neural Networks.

[5]  K. A. Semendyayev,et al.  Handbook of mathematics , 1985 .

[6]  T. Sejnowski.  Higher‐order Boltzmann machines , 1987 .

[7]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[8]  H Takayama,et al.  Spin glass properties of a class of mean-field models , 1990 .

[9]  Neil D. Lawrence,et al.  Mixture Representations for Inference and Learning in Boltzmann Machines , 1998, UAI.

[10]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[11]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[12]  Hilbert J. Kappen,et al.  Efficient Learning in Boltzmann Machines Using Linear Response Theory , 1998, Neural Computation.

[13]  Carsten Peterson,et al.  A Mean Field Theory Learning Algorithm for Neural Networks , 1987, Complex Syst..

[14]  S. Kirkpatrick,et al.  Solvable Model of a Spin-Glass , 1975 .

[15]  Hilbert J. Kappen,et al.  Boltzmann Machine Learning Using Mean Field Theory and Linear Response Correction , 1997, NIPS.

[16]  Manuel Graña,et al.  The high-order Boltzmann machine: learned distribution and topology , 1995, IEEE Trans. Neural Networks.

[17]  Geoffrey E. Hinton,et al.  Learning and relearning in Boltzmann machines , 1986 .