An Auxiliary Variational Method

An attractive feature of variational methods used in the context of approximate inference in undirected graphical models is a rigorous lower bound on the normalization constants. Here we explore the idea of using augmented variable spaces to improve on the standard mean-field bounds. Our approach forms a more powerful class of approximations than any structured mean field technique. Moreover, the existing variational mixture models may be seen as computationally expensive special cases of our method. A byproduct of our work is an efficient way to calculate a set of mixture coefficients for any set of tractable distributions that principally improves on a flat combination.

[1]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[2]  Wang,et al.  Nonuniversal critical dynamics in Monte Carlo simulations. , 1987, Physical review letters.

[3]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[4]  Michael I. Jordan,et al.  Computing upper and lower bounds on likelihoods in intractable networks , 1996, UAI.

[5]  Neil D. Lawrence,et al.  Approximating Posterior Distributions in Belief Networks Using Mixtures , 1997, NIPS.

[6]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[7]  D. Higdon.  Auxiliary Variable Methods for Markov Chain Monte Carlo with Applications , 1998 .

[8]  Michael I. Jordan,et al.  Improving the Mean Field Approximation Via the Use of Mixture Distributions , 1999, Learning in Graphical Models.

[9]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[10]  Neil D. Lawrence,et al.  Mixture Representations for Inference and Learning in Boltzmann Machines , 1998, UAI.

[11]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine-mediated learning.

[12]  David Barber,et al.  Tractable Variational Structures for Approximating Graphical Models , 1998, NIPS.

[13]  David Barber,et al.  Gaussian Fields for Approximate Inference in Layered Sigmoid Belief Networks , 1999, NIPS.

[14]  Nir Friedman,et al.  Incorporating Expressive Graphical Models in VariationalApproximations: Chain-graphs and Hidden Variables , 2001, UAI.

[15]  F. Agakov,et al.  Temporal Hidden Hopfield Models , 2002 .

[16]  David Barber,et al.  Approximate Learning in Temporal Hidden Hopfield Models , 2003, ICANN.

[17]  Michael I. Jordan,et al.  Factorial Hidden Markov Models , 1995, Machine Learning.

[18]  Martin J. Wainwright,et al.  A new class of upper bounds on the log partition function , 2002, IEEE Transactions on Information Theory.