The limitations of deterministic Boltzmann machine learning

The stochastic Boltzmann machine (SBM) learning procedure allows a system of stochastic binary units at thermal equilibrium to model arbitrary probabilistic distributions of binary vectors, but the inefficiency inherent in stochastic simulations limits its usefulness. By employing mean field theory, the stochastic settling to thermal equilibrium can be replaced by efficient deterministic settling to a steady state. The analogous deterministic Boltzmann machine (DBM) learning rule performs steepest descent in an appropriately defined error measure under certain circumstances and has been empirically shown to solve a variety of non-trivial supervised, input-output problems.However, by applying ‘naive’ mean field theory to a finite system with non-random interactions, the true stochastic system is not well described, and representational problems result that significantly limit the situations in which the DBM procedure can be successfully applied. It is shown that the independence assumption is unacceptably ...

[1]  R. Glauber.  Time‐Dependent Statistics of the Ising Model , 1963 .

[2]  R. Palmer,et al.  Solution of 'Solvable model of a spin glass' , 1977 .

[3]  A. Bray,et al.  Evidence for massless modes in the 'solvable model' of a spin glass , 1979 .

[4]  B. Huberman,et al.  Digital dynamics and the simulation of magnetic systems , 1983 .

[5]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[6]  H. Takayama,et al.  TAP free energy structure of SK spin glasses , 1985 .

[7]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[8]  Carsten Peterson,et al.  A Mean Field Theory Learning Algorithm for Neural Networks , 1987, Complex Syst..

[9]  J J Hopfield,et al.  Learning algorithms and probability distributions in feed-forward and feed-back networks. , 1987, Proceedings of the National Academy of Sciences of the United States of America.

[10]  Geoffrey E. Hinton.  Deterministic Boltzmann Learning Performs Steepest Descent in Weight-Space , 1989, Neural Computation.

[11]  Geoffrey E. Hinton,et al.  Learning distributed representations of concepts. , 1989 .

[12]  Waugh,et al.  Associative memory in an analog iterated-map neural network. , 1990, Physical review. A, Atomic, molecular, and optical physics.

[13]  F. R. Waugh,et al.  Fixed-point attractors in analog neural computation. , 1990, Physical review letters.

[14]  Carsten Peterson,et al.  Mean Field Theory Neural Networks for Feature Recognition, Content Addressable Memory and Optimization , 1991 .

[15]  David Haussler,et al.  Unsupervised learning of distributions on binary vectors using two layer networks , 1991, NIPS 1991.

[16]  Pierre Baldi,et al.  Contrastive Learning and Neural Oscillations , 1991, Neural Computation.

[17]  Javier R. Movellan,et al.  Contrastive Hebbian Learning in the Continuous Hopfield Model , 1991 .

[18]  Geoffrey E. Hinton,et al.  Deterministic Boltzmann Learning in Networks with Asymmetric Connectivity , 1991 .

[19]  Conrad Galland,et al.  Learning in Deterministic Boltzmann Machine Networks , 1992 .

[20]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.