Bootstrapping Likelihood for Model Selection with Small Samples

Abstract Akaike's information criterion (AIC), derived from asymptotics of the maximum likelihood estimator, is widely used in model selection. However, it has a finite-sample bias that produces overfitting in linear regression. To deal with this problem, Ishiguro, Sakamoto, and Kitagawa proposed a bootstrap-based extension to AIC which they called EIC. This article compares model-selection performance of AIC, EIC, a bootstrap-smoothed likelihood cross-validation (BCV) and its modification (632CV) in small-sample linear regression, logistic regression, and Cox regression. Simulation results show that EIC largely overcomes AIC's overfitting problem and that BCV may be better than EIC. Hence, the three methods based on bootstrapping the likelihood establish themselves as important alternatives to AIC in model selection with small samples.

[1]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[2]  S. T. Buckland,et al.  Computer Intensive Statistical Methods: Validation, Model Selection, and Bootstrap , 1993 .

[3]  D. Cox.  Regression Models and Life-Tables , 1972 .

[4]  N. L. Johnson,et al.  Breakthroughs in Statistics, Vol. III , 1998 .

[5]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[6]  G. Kitagawa,et al.  Bootstrapping Log Likelihood and EIC, an Extension of AIC , 1997 .

[7]  John A. Nelder,et al.  Generalized linear models. 2nd ed. , 1993 .

[8]  P. McCullagh,et al.  Generalized Linear Models , 1992 .

[9]  B. Efron.  Bootstrap Methods: Another Look at the Jackknife , 1979 .

[10]  Genshiro Kitagawa,et al.  Selected papers of Hirotugu Akaike , 1998 .

[11]  J. S. Urban Hjorth,et al.  Computer Intensive Statistical Methods: Validation, Model Selection, and Bootstrap , 1993 .

[12]  J. W. Gorman,et al.  Fitting Equations to Data. , 1973 .

[13]  L. Breiman.  Heuristics of instability and stabilization in model selection , 1996 .

[14]  Alan J. Miller.  Subset Selection in Regression , 1992 .

[15]  P. J. Huber.  The behavior of maximum likelihood estimates under nonstandard conditions , 1967 .

[16]  L. Breiman,et al.  Submodel selection and evaluation in regression. The X-random case , 1992 .

[17]  B. Efron.  Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation , 1983 .

[18]  P. McCullagh,et al.  Generalized Linear Models , 1984 .

[19]  R. Tibshirani,et al.  Improvements on Cross-Validation: The 632+ Bootstrap Method , 1997 .

[20]  David R. Cox,et al.  Regression models and life tables (with discussion , 1972 .

[21]  B. Silverman.  Density estimation for statistics and data analysis , 1986 .

[22]  C. L. Mallows.  Some comments on C_p , 1973 .

[23]  Clifford M. Hurvich,et al.  Regression and time series model selection in small samples , 1989 .

[24]  J. Kalbfleisch,et al.  The Statistical Analysis of Failure Time Data , 1980 .

[25]  M. Stone.  An Asymptotic Equivalence of Choice of Model by Cross‐Validation and Akaike's Criterion , 1977 .

[26]  N. L. Johnson,et al.  Breakthroughs in Statistics , 1992 .

[27]  A. McQuarrie,et al.  Regression and Time Series Model Selection , 1998 .

[28]  Alan J. Miller,et al.  Subset Selection in Regression , 1991 .

[29]  N. Sugiura.  Further analysts of the data by akaike' s information criterion and the finite corrections , 1978 .

[30]  David M. Allen,et al.  The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction , 1974 .

[31]  N. Draper,et al.  Applied Regression Analysis , 1966 .

[32]  Rupert G. Miller,et al.  Survival Analysis , 2022, The SAGE Encyclopedia of Research Design.

[33]  J. Shao.  Linear Model Selection by Cross-validation , 1993 .

[34]  G. McLachlan.  Discriminant Analysis and Statistical Pattern Recognition , 1992 .