Minimum Description Length Model Selection Criteria for Generalized Linear Models

This paper derives several model selection criteria for generalized linear models (GLMs) following the principle of Minimum Description Length (MDL). We focus our attention on the mixture form of MDL. Normal or normal-inverse gamma distributions are used to construct the mixtures, depending on whether or not we choose to account for possible over-dispersion in the data. In the latter case, we apply Efron's [6] double exponential family characterization of GLMs. Standard Laplace approximations are then employed to derive computationally tractable selection rules. Each constructed criterion has adaptive penalties on model complexity, either explicitly or implicitly. Theoretical results for the normal linear model, and a set of simulations for logistic regression, illustrate that mixture MDL can "bridge" the selection "extremes" AIC and BIC in the sense that it can mimic the performance of either criterion, depending on which is best for the situation at hand.

[1]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[2]  A. W. Kemp,et al.  Kendall's Advanced Theory of Statistics. , 1994 .

[3]  E. George,et al.  Journal of the American Statistical Association is currently published by American Statistical Association. , 2007 .

[4]  D. Freedman,et al.  How Many Variables Should Be Entered in a Regression Equation , 1983 .

[5]  Hirotugu Akaike.  An objective use of Bayesian models , 1977 .

[6]  R. Shibata.  An optimal selection of regression variables , 1981 .

[7]  Dean Phillips Foster,et al.  Calibration and empirical Bayes variable selection , 2000 .

[8]  Jorma Rissanen,et al.  Fisher information and stochastic complexity , 1996, IEEE Trans. Inf. Theory.

[9]  D. Spiegelhalter,et al.  Bayes Factors and Choice Criteria for Linear Models , 1980 .

[10]  A. Raftery.  Approximate Bayes factors and accounting for model uncertainty in generalised linear models , 1996 .

[11]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[12]  H. Akaike.  A new look at the statistical model identification , 1974 .

[13]  B. Efron.  Double Exponential Families and Their Use in Generalized Linear Regression , 1986 .

[14]  Bin Yu,et al.  Model Selection and the Principle of Minimum Description Length , 2001 .

[15]  L. Wasserman,et al.  Computing Bayes Factors by Combining Simulation and Asymptotic Approximations , 1997 .

[16]  Jorma Rissanen,et al.  The Minimum Description Length Principle in Coding and Modeling , 1998, IEEE Trans. Inf. Theory.

[17]  T. Speed,et al.  Model selection and prediction: Normal regression , 1993 .

[18]  A note on some model selection criteria , 1986 .