Recent advances in computing the NML for discrete Bayesian networks

Bayesian networks are parametric models for multidimensional domains exhibiting complex dependencies between the dimensions (domain variables). A central problem in learning such models is how to regularize the number of parameters; in other words, how to determine which dependencies are significant and which are not. The normalized maximum likelihood (NML) distribution or code offers an information-theoretic solution to this problem. Unfortunately, computing it for arbitrary Bayesian network models appears to be computationally infeasible, but recent results have showed that it can be computed efficiently for certain restricted type of Bayesian network models. In this review paper we summarize the main results. 1. NORMALIZED MAXIMUM LIKELIHOOD Let

[1]  Henry Tirri,et al.  On predictive distributions and Bayesian networks , 2000, Stat. Comput..

[2]  Tomi Silander,et al.  Factorized normalized maximum likelihood criterion for learning Bayesian network structures , 2008 .

[3]  A. Hasman,et al.  Probabilistic reasoning in intelligent systems: Networks of plausible inference , 1991 .

[4]  Jorma Rissanen,et al.  Efficient Computation of Stochastic Complexity , 2003 .

[5]  Henry Tirri,et al.  On Bayesian Case Matching , 1998, EWCBR.

[6]  Hannes Wettig,et al.  NML Computation Algorithms for Tree-Structured Multinomial Bayesian Networks , 2008, EURASIP J. Bioinform. Syst. Biol..

[7]  Michael I. Jordan.  Graphical Models , 2003 .

[8]  T. Roos,et al.  Bayesian network structure learning using factorized NML universal models , 2008, 2008 Information Theory and Applications Workshop.

[9]  Jorma Rissanen,et al.  Fisher information and stochastic complexity , 1996, IEEE Trans. Inf. Theory.

[10]  Petri Myllymäki,et al.  Computing the Multinomial Stochastic Complexity in Sub-Linear Time , 2008 .

[11]  Teemu Roos.  Monte Carlo estimation of minimax regret with an application to MDL model selection , 2008, 2008 IEEE Information Theory Workshop.

[12]  Petri Myllymäki,et al.  A linear-time algorithm for computing the multinomial stochastic complexity , 2007, Inf. Process. Lett..

[13]  P. Myllymaki,et al.  On recurrence formulas for computing the stochastic complexity , 2008, 2008 International Symposium on Information Theory and Its Applications.

[14]  Petri Myllymäki,et al.  Fast NML Computation for Naive Bayes Models , 2007, Discovery Science.

[15]  D. Knuth,et al.  A recurrence related to trees , 1989 .

[16]  Jorma Rissanen,et al.  An MDL Framework for Data Clustering , 2005 .

[17]  Jorma Rissanen,et al.  MDL Denoising , 2000, IEEE Trans. Inf. Theory.

[18]  Henry Tirri,et al.  Supervised model-based visualization of high-dimensional data , 2000, Intell. Data Anal..

[19]  Hannes Wettig,et al.  Calculating the Normalized Maximum Likelihood Distribution for Bayesian Forests , 2007 .

[20]  P. Grünwald,et al.  An empirical study of minimum description length model selection with infinite parametric complexity , 2006 .

[21]  Mikko Koivisto,et al.  Parent Assignment Is Hard for the MDL, AIC, and NML Costs , 2006, COLT.

[22]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[23]  Henry Tirri,et al.  Factorized NML models , 2008 .

[24]  Petri Myllymäki,et al.  On the Multinomial Stochastic Complexity and its Connection to the Birthday Problem , 2008, ITSL.

[25]  Petri Myllymäki,et al.  A Fast Normalized Maximum Likelihood Algorithm for Multinomial Data , 2005, IJCAI.

[26]  Henry Tirri,et al.  Minimum Encoding Approaches for Predictive Modeling , 1998, UAI.

[27]  Gaston H. Gonnet,et al.  On the LambertW function , 1996, Adv. Comput. Math..

[28]  Petri Myllymäki,et al.  Computing the NML for Bayesian forests via matrices and generating polynomials , 2008, 2008 IEEE Information Theory Workshop.

[29]  Mikko Koivisto,et al.  Sum-Product Algorithms for the Analysis of Genetic Risks , 2004 .