Model Selection with Information Criteria

This thesis is on model selection using information criteria. The information criteria include generalized information criterion and a family of Bayesian information criteria. The properties and improvement of the information criteria are investigated. We analyze nonasymptotic and asymptotic properties of the information criteria for linear models, probabilistic models, and high dimensional models, respectively. We give probability of selecting a model and compute the probability by Monte Carlo methods. We derive the conditions under which the criteria are overfitting, consistent, or underfitting. We further propose new model selection procedures to improve the information criteria. The procedures combine the information criteria with the probability of selecting a model and overfitting level, respectively. In addition, we develop model selection software packages in R and examine applications to real data.

[1]  Jorma Rissanen,et al.  Information and Complexity in Statistical Modeling , 2006, ITW.

[2]  A. I. McLeod,et al.  Parsimony, model adequacy and periodic correlation in time series forecasting , 1993, 1611.01535.

[3]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[4]  Yongil Jeon,et al.  Forecasting Performance of Information Criteria with Many Macro Series , 2004 .

[5]  D. Madigan,et al.  [Least Angle Regression]: Discussion , 2004 .

[6]  David R. Anderson,et al.  Model Selection and Multimodel Inference , 2003 .

[7]  Dean P. Foster,et al.  The risk inflation criterion for multiple regression , 1994 .

[8]  Jonnagadda S Rao,et al.  Bootstrap choice of cost complexity for better subset selection , 1999 .

[9]  R. J. Bhansali,et al.  A derivation of the information criteria for selecting autoregressive models , 1986, Advances in Applied Probability.

[10]  Steven Kay,et al.  Modern Spectral Estimation: Theory and Application , 1988 .

[11]  R. Nishii.  Asymptotic Properties of Criteria for Selection of Variables in Multiple Regression , 1984 .

[12]  H. Akaike.  Statistical predictor identification , 1970 .

[13]  Yuhong Yang.  Can the Strengths of AIC and BIC Be Shared , 2005 .

[14]  A. I. McLeod,et al.  bestglm: Best Subset GLM , 2009 .

[15]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[16]  H. Akaike.  A new look at the statistical model identification , 1974 .

[17]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[18]  Robert W. Wilson,et al.  Regressions by Leaps and Bounds , 2000, Technometrics.

[19]  J. Shao.  AN ASYMPTOTIC THEORY FOR LINEAR MODEL SELECTION , 1997 .

[20]  Clifford M. Hurvich,et al.  Regression and time series model selection in small samples , 1989 .

[21]  C. Mallows.  More comments on C p , 1995 .

[22]  Yongli Zhang,et al.  Model selection: A Lagrange optimization approach , 2009 .

[23]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[24]  H. White,et al.  Information criteria for selecting possibly misspecified parametric models , 1996 .

[25]  J. Shao.  Linear Model Selection by Cross-validation , 1993 .

[26]  J. Friedman,et al.  A Statistical View of Some Chemometrics Regression Tools , 1993 .

[27]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory, Second Edition , 2000, Statistics for Engineering and Information Science.

[28]  J. Shao.  Bootstrap Model Selection , 1996 .

[29]  R. Shibata.  Approximate efficiency of a selection procedure for the number of regression variables , 1984 .

[30]  T. Hastie,et al.  [A Statistical View of Some Chemometrics Regression Tools]: Discussion , 1993 .

[31]  M. Kupperman.  Linear Statistical Inference and Its Applications 2nd Edition (C. Radhakrishna Rao) , 1975 .

[32]  M. Peruggia.  Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach (2nd ed.) , 2003 .

[33]  H. Zou.  The Adaptive Lasso and Its Oracle Properties , 2006 .

[34]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[35]  Xiaotong Shen,et al.  Adaptive Model Selection , 2002 .

[36]  A. McQuarrie,et al.  Regression and Time Series Model Selection , 1998 .

[37]  Jianqing Fan,et al.  A Selective Overview of Variable Selection in High Dimensional Feature Space. , 2009, Statistica Sinica.

[38]  William S. Cleveland,et al.  The Inverse Autocorrelations of a Time Series and Their Applications , 1972 .

[39]  Jun Shao,et al.  The gic for model selection : a hypothesis testing approach , 2000 .

[40]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[41]  R. Tibshirani,et al.  PATHWISE COORDINATE OPTIMIZATION , 2007, 0708.1485.

[42]  R. H. Myers.  Classical and modern regression with applications , 1986 .

[43]  Christian P. Robert,et al.  The Bayesian choice : from decision-theoretic foundations to computational implementation , 2007 .

[44]  Marc Hofmann,et al.  Efficient algorithms for computing the best subset regression models for large-scale problems , 2007, Comput. Stat. Data Anal..

[45]  A. Atkinson.  A note on the generalized information criterion for choice of a model , 1980 .

[46]  Runze Li,et al.  Tuning parameter selectors for the smoothly clipped absolute deviation method. , 2007, Biometrika.

[47]  李幼升,et al.  Ph , 1989 .

[48]  Calyampudi Radhakrishna Rao,et al.  Linear Statistical Inference and its Applications , 1967 .

[49]  Howell Tong,et al.  Some Comments on the Canadian Lynx Data , 1977 .

[50]  Dean Phillips Foster,et al.  Calibration and Empirical Bayes Variable Selection , 1997 .

[51]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[52]  Jiahua Chen,et al.  Extended Bayesian information criteria for model selection with large model spaces , 2008 .

[53]  Anil K. Bera.  Hypothesis Testing in the 20th Century with a Special Reference to Testing with Misspecified Models , 1999 .

[54]  A. Laupacis,et al.  Rates of Hyperkalemia after Publication of the Randomized Aldactone Evaluation Study , 2004 .

[55]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[56]  C. Mallows.  Some Comments on Cp , 2000, Technometrics.

[57]  H. Akaike.  Fitting autoregressive models for prediction , 1969 .

[58]  Bin Yu,et al.  Model Selection and the Principle of Minimum Description Length , 2001 .

[59]  Christian P. Robert,et al.  Estimation of noncentrality parameters , 1993 .

[60]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[61]  Erricos John Kontoghiorghes,et al.  A branch and bound algorithm for computing the best subset regression models , 2002 .

[62]  Leo Breiman,et al.  Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author) , 2001, Statistical Science.

[63]  Donald B. Percival,et al.  Spectral Analysis for Physical Applications , 1993 .

[64]  R. Bhansali,et al.  Some properties of the order of an autoregressive model selected by a generalization of Akaike∘s EPF criterion , 1977 .

[65]  Runze Li,et al.  Regularization Parameter Selections via Generalized Information Criterion , 2010, Journal of the American Statistical Association.

[66]  Walter Zucchini,et al.  Model Selection , 2011, International Encyclopedia of Statistical Science.

[67]  Alan J. Miller.  Subset Selection in Regression , 1992 .

[68]  K. Hipel,et al.  Time series modelling of water resources and environmental systems , 1994 .

[69]  R. Tibshirani,et al.  The Covariance Inflation Criterion for Adaptive Model Selection , 1999 .

[70]  J. Rissanen.  A UNIVERSAL PRIOR FOR INTEGERS AND ESTIMATION BY MINIMUM DESCRIPTION LENGTH , 1983 .

[71]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[72]  H. Akaike.  A Bayesian extension of the minimum AIC procedure of autoregressive model fitting , 1979 .

[73]  L. Breiman.  Heuristics of instability and stabilization in model selection , 1996 .