Properties of Kikuchi approximations constructed from clique based decompositions

Kikuchi approximations constructed from clique-based decompositions can be used to calculate suitable approximations of probability distributions. They can be applied in domains such as probabilistic modeling, supervised and unsupervised classification, and evolutionary algorithms. This paper introduces a number of properties of these approximations. Pairwise and local Markov properties of the Kikuchi approximations are proved. We prove that, even if the global Markov property is not satisfied in the general case, it is possible to decompose the Kikuchi approximation in the product of local Kikuchi approximations defined on a decomposition of the graph. Partial Kikuchi approximations are introduced. Additionally, the paper clarifies the place of clique-based decompositions in relation to other techniques inspired by methods from statistical physics, and discusses the application of the results introduced in the paper for the conception of Kikuchi approximation learning algorithms.

[1]  Yee Whye Teh,et al.  On Improving the Efficiency of the Iterative Proportional Fitting Procedure , 2003, AISTATS.

[2]  William T. Freeman,et al.  Understanding belief propagation and its generalizations , 2003 .

[3]  Robin Höns,et al.  Estimation of distribution algorithms and minimum relative entropy , 2005 .

[4]  S. Aji,et al.  The Generalized Distributive Law and Free Energy Minimization , 2001 .

[5]  Robert J. McEliece,et al.  The generalized distributive law , 2000, IEEE Trans. Inf. Theory.

[6]  Sekhar Tatikonda,et al.  Loopy Belief Propogation and Gibbs Measures , 2002, UAI.

[7]  Avi Pfeffer,et al.  Loopy Belief Propagation as a Basis for Communication in Sensor Networks , 2002, UAI.

[8]  Jessika Weiss,et al.  Graphical Models In Applied Multivariate Statistics , 2016 .

[9]  A. Yuille.  A Double-Loop Algorithm to Minimize the Bethe and Kikuchi Free Energies , 2001 .

[10]  Joachim Rosenthal,et al.  Mathematical systems theory in biology, communications, computation, and finance , 2003 .

[11]  T. Morita.  Formal Structure of the Cluster Variation Method , 1994 .

[12]  J. A. Lozano,et al.  Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation , 2001 .

[13]  L. Sevastianov,et al.  THE SUPERPOSITION APPROXIMATION: A CRITICAL REVIEW , 2004 .

[14]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[15]  Max Welling,et al.  On the Choice of Regions for Generalized Belief Propagation , 2004, UAI.

[16]  X. Jin.  Factor graphs and the Sum-Product Algorithm , 2002 .

[17]  Kristian G. Olesen,et al.  Maximal Prime Subgraph Decomposition of Bayesian Networks , 2001, FLAIRS.

[18]  T. Heskes.  Stable Fixed Points of Loopy Belief Propagation Are Minima of the Bethe Free Energy , 2002 .

[19]  D. Nilsson,et al.  An efficient algorithm for finding the M most probable configurationsin probabilistic expert systems , 1998, Stat. Comput..

[20]  Wainwrighty Tommi Jaakkola,et al.  Tree-based reparameterization framework for analysis of beliefpropagation and related algorithmsMartin , 2001 .

[21]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[22]  R. Kikuchi.  A Theory of Cooperative Phenomena , 1951 .

[23]  INFORMATION CONTENT OF SIGNALS USING CORRELATION FUNCTION EXPANSIONS OF THE ENTROPY , 1997 .

[24]  Roberto Santana,et al.  Estimation of Distribution Algorithms with Kikuchi Approximations , 2005, Evolutionary Computation.

[25]  Qiang Hu,et al.  Critical Remarks on the Maximal Prime Decomposition of Bayesian Networks , 2003, RSFDGrC.

[26]  Tom Minka,et al.  A family of algorithms for approximate Bayesian inference , 2001 .

[27]  J. Kirkwood,et al.  The Radial Distribution Function in Liquids , 1942 .

[28]  I. Bratko,et al.  Kikuchi-Bayes: Factorized Models for Approximate Classification in Closed Form , 2004 .

[29]  H. Bethe.  Statistical Theory of Superlattices , 1935 .

[30]  Robert E. Tarjan,et al.  Decomposition by clique separators , 1985, Discret. Math..

[31]  Francesco M. Malvestuto,et al.  An implementation of the iterative proportional fitting procedure by propagation trees , 2001 .

[32]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[33]  Matsuda,et al.  Physical nature of higher-order mutual information: intrinsic correlations and frustration , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[34]  Ivan Bratko,et al.  Testing the significance of attribute interactions , 2004, ICML.

[35]  Michael I. Jordan,et al.  Graph Partition Strategies for Generalized Mean Field Inference , 2004, UAI.

[36]  T. Morita.  Cluster Variation Method for Non-Uniform Ising and Heisenberg Models and Spin-Pair Correlation Function , 1991 .