BOOTSTRAP ESTIMATE OF KULLBACK-LEIBLER INFORMATION FOR MODEL SELECTION

Estimation of Kullback-Leibler information is a crucial part of deriving a statistical model selection procedure which, like AIC, is based on the likelihood principle. To discriminate between nested models, we have to estimate Kullback- Leibler information up to the order of a constant, while Kullback-Leibler informa- tion itself is of the order of the number of observations. A correction term employed in AIC is an example of how to fulfill this requirement; however the correction is a simple minded bias correction to the log maximum likelihood and there is no assurance that such a bias correction yields a good estimate of Kullback-Leibler information. In this paper we investigate a bootstrap type estimate of Kullback- Leibler information as an alternative. We first show that both bootstrap estimates proposed by Efron (1983, 1986) and by Cavanaugh and Shumway (1997) are at least asymptotically equivalent and there exist many other equivalent bootstrap estimates. We also show that all such methods are asymptotically equivalent to a non-bootstrap method known as TIC(Takeuchi (1976)), which is a generalization of AIC when the re-sampling method is non-parametric. Otherwise, for example, if the re-sampling method is parametric they are asymptotically equivalent to AIC. Therefore, the use of a bootstrap type estimate is not advantageous if enough ob- servations are available and simple calculations of a non-bootstrap estimate AIC or TIC is not a burden. At the same time, it is also true that the use of a bootstrap estimate in place of a non-bootstrap estimate is reasonable and advantageous if the non-bootstrap estimate is too complicated to evaluate analytically.

[1]  S. Zacks.  The theory of statistical inference , 1972 .

[2]  Shelemyahu Zacks,et al.  The Theory of Statistical Inference. , 1972 .

[3]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[4]  N. Sugiura.  Further analysts of the data by akaike' s information criterion and the finite corrections , 1978 .

[5]  Ryoichi Shimizu,et al.  Entropy maximization principle and selection of the order of an autoregressive Gaussian process , 1978 .

[6]  R. Shibata.  Asymptotically Efficient Selection of the Order of the Model for Estimating Parameters of a Linear Process , 1980 .

[7]  R. Shibata.  An optimal selection of regression variables , 1981 .

[8]  D. Freedman.  Bootstrapping Regression Models , 1981 .

[9]  B. Efron.  Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation , 1983 .

[10]  B. Efron.  How Biased is the Apparent Error Rate of a Prediction Rule , 1986 .

[11]  R. Shibata.  Statistical aspects of model selection , 1989 .

[12]  Clifford M. Hurvich,et al.  Regression and time series model selection in small samples , 1989 .

[13]  M. Ishiguro,et al.  Application of an estimator-free information criterion (WIC) to aperture synthesis imaging , 1991 .

[14]  D. Stoffer,et al.  Bootstrapping State-Space Models: Gaussian Maximum Likelihood Estimation and the Kalman Filter , 1991 .

[15]  Clifford M. Hurvich,et al.  Bias of the corrected AIC criterion for underfitted regression and time series models , 1991 .

[16]  Clifford M. Hurvich,et al.  A CORRECTED AKAIKE INFORMATION CRITERION FOR VECTOR AUTOREGRESSIVE MODEL SELECTION , 1993 .

[17]  J. Pfanzagl.  Parametric Statistical Theory , 1994 .

[18]  J. Cavanaugh,et al.  A BOOTSTRAP VARIANT OF AIC FOR STATE-SPACE MODEL SELECTION , 1997 .