On the Computation of Entropy Prior Complexity and Marginal Prior Distribution for the Bernoulli Model

As the size and complexity of models grow, the choice of the best model becomes a difficult and challenging task. Once the best model is specified, the goodness of fit of the model needs to be examined first. A highly complex model may provide a good fit, but giving no consideration to model complexity could result in incorrect estimates of parameter values and predictions. In order to improve the model selection process, model complexity needs to be defined clearly. This article studies different aspects of model complexity and discusses the extent to which they can be measured. The most common attribute that is usually ignored from many complexity measures is the parameter prior, which is an inherent part of the model and could impact the complexity significantly. The concept of parameter prior and its connection to model complexity are therefore discussed here, and some relationships to the entropy measure elements are also addressed.

[1]  A. Linde.  A Bayesian view of model complexity , 2012 .

[2]  N. Balakrishnan,et al.  Analysis of a supersaturated design using Entropy Prior Complexity for binary responses via generalized linear models , 2012 .

[3]  Jorma Rissanen,et al.  Optimal Estimation of Parameters , 2012 .

[4]  Ricardo López-Ruiz,et al.  A Statistical Measure of Complexity , 1995, ArXiv.

[5]  Wolf Vanpaemel,et al.  Measuring model complexity with the prior predictive , 2009, NIPS.

[6]  A. Caticha.  Information and Entropy , 2007, 0710.1068.

[7]  P. Grünwald.  The Minimum Description Length Principle (Adaptive Computation and Machine Learning) , 2007 .

[8]  Jorma Rissanen,et al.  Information and Complexity in Statistical Modeling , 2006, ITW.

[9]  M. Tribus,et al.  Probability theory: the logic of science , 2003 .

[10]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[11]  Charles S. Bos.  A Comparison of Marginal Likelihood Computation Methods , 2002, COMPSTAT.

[12]  Ricardo López-Ruiz,et al.  Features of the extension of a statistical measure of complexity to continuous systems. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[13]  Michael D. Lee.  Generating Additive Clustering Models with Minimal Stochastic Complexity , 2002, J. Classif..

[14]  Bin Yu,et al.  Model Selection and the Principle of Minimum Description Length , 2001 .

[15]  Naftali Tishby,et al.  Predictability, Complexity, and Learning , 2000, Neural Computation.

[16]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[17]  I. J. Myung,et al.  Counting probability distributions: Differential geometry and model selection , 2000, Proc. Natl. Acad. Sci. USA.

[18]  J. Dunn.  Model complexity: The fit to random data reconsidered , 2000, Psychological research.

[19]  I. J. Myung,et al.  The Importance of Complexity in Model Selection. , 2000, Journal of mathematical psychology.

[20]  M. C. Bueso,et al.  Stochastic complexity and model selection from incomplete data , 1999 .

[21]  J. Crutchfield,et al.  Measures of statistical complexity: Why? , 1998 .

[22]  Jorma Rissanen,et al.  Stochastic Complexity in Statistical Inquiry , 1989, World Scientific Series in Computer Science.

[23]  I. J. Myung,et al.  Applying Occam’s razor in modeling cognition: A Bayesian approach , 1997 .

[24]  V. Balasubramanian.  Statistical Inference, Occam's Razor, and Statistical Mechanics on the Space of Probability Distributions , 1996, Neural Computation.

[25]  John D. Lafferty,et al.  Inducing Features of Random Fields , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Roger J. Brooks,et al.  Choosing the best model: Level of detail, complexity, and model performance , 1996 .

[27]  Adam L. Berger,et al.  A Maximum Entropy Approach to Natural Language Processing , 1996, CL.

[28]  Jorma Rissanen,et al.  Fisher information and stochastic complexity , 1996, IEEE Trans. Inf. Theory.

[29]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 2019, Texts in Computer Science.

[30]  Young,et al.  Inferring statistical complexity. , 1989, Physical review letters.

[31]  J. Glenn Brookshear,et al.  Theory of Computation: Formal Languages, Automata, and Complexity , 1989 .

[32]  E. Hannan,et al.  On stochastic complexity and nonparametric density estimation , 1988 .

[33]  J. Rissanen.  Stochastic Complexity and Modeling , 1986 .

[34]  Charles H. Bennett,et al.  On the nature and origin of complexity in discrete, homogeneous, locally-interacting systems , 1986 .

[35]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .