A Tighter Bound for Graphical Models

We present a method to bound the partition function of a Boltzmann machine neural network with any odd-order polynomial. This is a direct extension of the mean-field bound, which is first order. We show that the third-order bound is strictly better than mean field. Additionally, we derive a third-order bound for the likelihood of sigmoid belief networks. Numerical experiments indicate that an error reduction of a factor of two is easily reached in the region where expansion-based approximations are useful.

[1]  S. Kirkpatrick,et al.  Solvable Model of a Spin-Glass , 1975 .

[2]  R. Palmer,et al.  Solution of 'Solvable model of a spin glass' , 1977 .

[3]  T. Plefka.  Convergence condition of the TAP equation for the infinite-ranged Ising spin glass model , 1982 .

[4]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[5]  Carsten Peterson,et al.  A Mean Field Theory Learning Algorithm for Neural Networks , 1987, Complex Syst..

[6]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems , 1988 .

[7]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[8]  Radford M. Neal.  Connectionist Learning of Belief Networks , 1992, Artif. Intell..

[9]  Michael I. Jordan,et al.  MASSACHUSETTS INSTITUTE OF TECHNOLOGY ARTIFICIAL INTELLIGENCE LABORATORY and CENTER FOR BIOLOGICAL AND COMPUTATIONAL LEARNING DEPARTMENT OF BRAIN AND COGNITIVE SCIENCES , 1996 .

[10]  Michael I. Jordan,et al.  Mean Field Theory for Sigmoid Belief Networks , 1996, J. Artif. Intell. Res..

[11]  Michael I. Jordan,et al.  Recursive Algorithms for Approximating Probabilities in Graphical Models , 1996, NIPS.

[12]  Michael I. Jordan,et al.  Computing upper and lower bounds on likelihoods in intractable networks , 1996, UAI.

[13]  Hilbert J. Kappen,et al.  Boltzmann Machine Learning Using Mean Field Theory and Linear Response Correction , 1997, NIPS.

[14]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[15]  David Barber,et al.  Tractable Undirected Approximations for Graphical Models , 1998 .

[16]  David Barber,et al.  Tractable Variational Structures for Approximating Graphical Models , 1998, NIPS.

[17]  Hilbert J. Kappen,et al.  Validity of TAP equations in neural networks , 1999 .

[18]  Martin J. Wainwright,et al.  Tree-based reparameterization framework for analysis of sum-product and related algorithms , 2003, IEEE Trans. Inf. Theory.