Learning loosely connected Markov random fields

We consider the structure learning problem for graphical models that we call loosely connected Markov random fields, in which the number of short paths between any pair of nodes is small, and present a new conditional independence test based algorithm for learning the underlying graph structure. The novel maximization step in our algorithm ensures that the true edges are detected correctly even when there are short cycles in the graph. The number of samples required by our algorithm is C log p, where p is the size of the graph and the constant C depends on the parameters of the model. We show that several previously studied models are examples of loosely connected Markov random fields, and our algorithm achieves the same or lower computational complexity than the previously designed algorithms for individual cases. We also get new results for more general graphical models, in particular, our algorithm learns general Ising models on the Erdős-Renyi random graph 𝒢(p,cp) correctly with running time O(np5).

[1]  D. Walkup,et al.  Association of Random Variables, with Applications , 1967 .

[2]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[3]  Noga Alon,et al.  The Probabilistic Method , 2015, Fundamentals of Ramsey Theory.

[4]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[5]  Elchanan Mossel,et al.  Glauber dynamics on trees and hyperbolic graphs , 2001, Proceedings 2001 IEEE International Conference on Cluster Computing.

[6]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[7]  Dror Weitz,et al.  Counting independent sets up to the tree threshold , 2006, STOC '06.

[8]  Kyomin Jung,et al.  Local approximate inference algorithms , 2006, ArXiv.

[9]  Alexandre d'Aspremont,et al.  Model Selection Through Sparse Max Likelihood Estimation Model Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data , 2022 .

[10]  Elchanan Mossel,et al.  Reconstruction of Markov Random Fields from Samples: Some Observations and Algorithms , 2007, SIAM J. Comput..

[11]  Elchanan Mossel,et al.  Rapid mixing of Gibbs sampling on graphs that are sparse on average , 2007, SODA '08.

[12]  Andrea Montanari,et al.  Which graphical models are difficult to learn? , 2009, NIPS.

[13]  J. Lafferty,et al.  High-dimensional Ising model selection using ℓ1-regularized logistic regression , 2010, 1010.0311.

[14]  Sanjay Shakkottai,et al.  Greedy learning of Markov network structure , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[15]  Thomas M Liggett,et al.  Stochastic models for large interacting systems and related correlation inequalities , 2010, Proceedings of the National Academy of Sciences.

[16]  Vincent Y. F. Tan,et al.  High-Dimensional Structure Estimation in Ising Models: Tractable Graph Families , 2011, ArXiv.

[17]  Fengshan Bai,et al.  Approximating partition functions of the two-state spin system , 2011, Inf. Process. Lett..

[18]  Sanjay Shakkottai,et al.  Greedy learning of graphical models with small girth , 2012, 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[19]  Martin J. Wainwright,et al.  Information-Theoretic Limits of Selecting Binary Graphical Models in High Dimensions , 2009, IEEE Transactions on Information Theory.

[20]  Peter Buhlmann,et al.  Geometry of the faithfulness assumption in causal inference , 2012, 1207.0547.

[21]  Todd P. Coleman,et al.  Directed Information Graphs , 2012, IEEE Transactions on Information Theory.