Information Theoretic Properties of Markov Random Fields, and their Algorithmic Applications

Markov random fields are a popular model for high-dimensional probability distributions. Over the years, many mathematical, statistical and algorithmic problems on them have been studied. Until recently, the only known algorithms for provably learning them relied on exhaustive search, correlation decay or various incoherence assumptions. Bresler gave an algorithm for learning general Ising models on bounded degree graphs. His approach was based on a structural result about mutual information in Ising models. Here we take a more conceptual approach to proving lower bounds on the mutual information. Our proof generalizes well beyond Ising models, to arbitrary Markov random fields with higher order interactions. As an application, we obtain algorithms for learning Markov random fields on bounded degree graphs on $n$ nodes with $r$-order interactions in $n^r$ time and $\log n$ sample complexity. Our algorithms also extend to various partial observation models.

[1]  Imre Csiszár,et al.  Consistent estimation of the basic neighborhood of Markov random fields , 2004, International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings..

[2]  Maria-Florina Balcan,et al.  Active Learning Algorithms for Graphical Model Selection , 2016, AISTATS.

[3]  J. Lafferty,et al.  High-dimensional Ising model selection using ℓ1-regularized logistic regression , 2010, 1010.0311.

[4]  Allan Sly,et al.  Computational Transition at the Uniqueness Threshold , 2010, 2010 IEEE 51st Annual Symposium on Foundations of Computer Science.

[5]  Elchanan Mossel,et al.  On the hardness of sampling independent sets beyond the tree threshold , 2007, math/0701471.

[6]  Raghu Meka,et al.  Learning Graphical Models Using Multiplicative Weights , 2017, 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS).

[7]  C. N. Liu,et al.  Approximating discrete probability distributions with dependence trees , 1968, IEEE Trans. Inf. Theory.

[8]  Martin J. Wainwright,et al.  Information-Theoretic Limits of Selecting Binary Graphical Models in High Dimensions , 2009, IEEE Transactions on Information Theory.

[9]  T. Sanders,et al.  Analysis of Boolean Functions , 2012, ArXiv.

[10]  Gregory Valiant,et al.  Finding Correlations in Subquadratic Time, with Applications to Learning Parities and Juntas , 2012, 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science.

[11]  Éva Tardos,et al.  Approximation algorithms for classification problems with pairwise relationships: metric labeling and Markov random fields , 2002, JACM.

[12]  S. Brush.  History of the Lenz-Ising Model , 1967 .

[13]  Elchanan Mossel,et al.  Reconstruction of Markov Random Fields from Samples: Some Observations and Algorithms , 2007, SIAM J. Comput..

[14]  Nathan Srebro,et al.  Maximum likelihood bounded tree-width Markov networks , 2001, Artif. Intell..

[15]  Daphne Koller,et al.  Efficient Structure Learning of Markov Networks using L1-Regularization , 2006, NIPS.

[16]  Pieter Abbeel,et al.  Learning Factor Graphs in Polynomial Time and Sample Complexity , 2006, J. Mach. Learn. Res..

[17]  Guy Bresler,et al.  Efficiently Learning Ising Models on Arbitrary Graphs , 2014, STOC.

[18]  Ali Jalali,et al.  On Learning Discrete Graphical Models using Group-Sparse Regularization , 2011, AISTATS.

[19]  Vincent Y. F. Tan,et al.  High-dimensional structure estimation in Ising models: Local separation criterion , 2011, 1107.1736.

[20]  Sanjoy Dasgupta,et al.  Learning Polytrees , 1999, UAI.

[21]  F. Martinelli,et al.  Approach to equilibrium of Glauber dynamics in the one phase region , 1994 .

[22]  Anima Anandkumar,et al.  Learning Mixtures of Tree Graphical Models , 2012, NIPS.

[23]  Michael Chertkov,et al.  Interaction Screening: Efficient and Sample-Optimal Learning of Ising Models , 2016, NIPS.

[24]  Allan Sly,et al.  The Computational Hardness of Counting in Two-Spin Models on d-Regular Graphs , 2012, 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science.