Optimization of Structured Mean Field Objectives

In intractable, undirected graphical models, an intuitive way of creating structured mean field approximations is to select an acyclic tractable subgraph. We show that the hardness of computing the objective function and gradient of the mean field objective qualitatively depends on a simple graph property. If the tractable subgraph has this property---we call such subgraphs v-acyclic---a very fast block coordinate ascent algorithm is possible. If not, optimization is harder, but we show a new algorithm based on the construction of an auxiliary exponential family that can be used to make inference possible in this case as well. We discuss the advantages and disadvantages of each regime and compare the algorithms empirically.

[1]  Gang Hua,et al.  Sequential mean field variational analysis of structured deformable shapes , 2006, Comput. Vis. Image Underst..

[2]  Michael I. Jordan,et al.  Variational inference in graphical models: The view from the marginal polytope , 2008 .

[3]  David Barber,et al.  Tractable Variational Structures for Approximating Graphical Models , 1998, NIPS.

[4]  Carsten Peterson,et al.  A Mean Field Theory Learning Algorithm for Neural Networks , 1987, Complex Syst..

[5]  Martin J. Wainwright,et al.  Log-determinant relaxation for approximate inference in discrete Markov random fields , 2006, IEEE Transactions on Signal Processing.

[6]  Nando de Freitas,et al.  Variational MCMC , 2001, UAI.

[7]  Yuan Qi,et al.  Tree-structured Approximations by Expectation Propagation , 2003, NIPS.

[8]  Wim Wiegerinck,et al.  Variational Approximations between Mean Field Theory and the Junction Tree Algorithm , 2000, UAI.

[9]  Jean B. Lasserre,et al.  Global Optimization with Polynomials and the Problem of Moments , 2000, SIAM J. Optim..

[10]  W. Freeman,et al.  Generalized Belief Propagation , 2000, NIPS.

[11]  Michael I. Jordan,et al.  Exploiting Tractable Substructures in Intractable Networks , 1995, NIPS.

[12]  Christopher Meek,et al.  A Variational Inference Procedure Allowing Internal Structure for Overlapping Clusters and Deterministic Constraints , 2006, J. Artif. Intell. Res..

[13]  Michael I. Jordan,et al.  Variational inference for Dirichlet process mixtures , 2006 .

[14]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[15]  Tommi S. Jaakkola,et al.  Approximate inference using planar graph decomposition , 2006, NIPS.