Cross-Validation Estimates IMSE

Integrated Mean Squared Error (IMSE) is a version of the usual mean squared error criterion, averaged over all possible training sets of a given size. If it could be observed, it could be used to determine optimal network complexity or optimal data subsets for efficient training. We show that two common methods of cross-validating average squared error deliver unbiased estimates of IMSE, converging to IMSE with probability one. These estimates thus make possible approximate IMSE-based choice of network complexity. We also show that two variants of cross validation measure provide unbiased IMSE-based estimates potentially useful for selecting optimal data subsets.

[1]  Halbert White,et al.  Estimation, inference, and specification analysis , 1996 .

[2]  Garrison W. Cottrell,et al.  Learning Mackey-Glass from 25 Examples, Plus or Minus 2 , 1993, NIPS.

[3]  Charles Elkan,et al.  Estimating the Accuracy of Learned Concepts , 1993, IJCAI.

[4]  Mark Plutowski,et al.  Selecting concise training sets from clean data , 1993, IEEE Trans. Neural Networks.

[5]  Yong Liu,et al.  Neural Network Model Selection Using Asymptotic Jackknife Estimator and Cross-Validation Method , 1992, NIPS.

[6]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[7]  Julian J. Faraway.  Sequential Design for the Nonparametric Regression of Curves and Surfaces , 1992 .

[8]  John E. Moody,et al.  The Effective Number of Parameters: An Analysis of Generalization and Regularization in Nonlinear Learning Systems , 1991, NIPS.

[9]  D. Andrews,et al.  Asymptotic optimality of generalized CL, cross-validation, and generalized cross-validation in regression with heteroskedastic errors , 1991 .

[10]  Halbert White,et al.  Learning in Artificial Neural Networks: A Statistical Perspective , 1989, Neural Computation.

[11]  M. Deaton,et al.  Response Surfaces: Designs and Analyses , 1989 .

[12]  George E. P. Box,et al.  Empirical Model‐Building and Response Surfaces , 1988 .

[13]  Ker-Chau Li,et al.  Asymptotic Optimality for $C_p, C_L$, Cross-Validation and Generalized Cross-Validation: Discrete Index Set , 1987 .

[14]  Florencio I. Utreras,et al.  On Generalized Cross-Validation for Multivariate Smoothing Spline Functions , 1987 .

[15]  James Stephen Marron,et al.  A Comparison of Cross-Validation Techniques in Density Estimation , 1987 .

[16]  Ker-Chau Li,et al.  Asymptotic optimality of CL and generalized cross-validation in ridge regression with application to spline smoothing , 1986 .

[17]  H. White.  Asymptotic theory for econometricians , 1985 .

[18]  C. J. Stone,et al.  An Asymptotically Optimal Window Selection Rule for Kernel Density Estimates , 1984 .

[19]  D. M. Titterington,et al.  Cross-validation in nonparametric estimation of probabilities and probability densities , 1984 .

[20]  A. Bowman.  An alternative method of cross-validation for the smoothing of density estimates , 1984 .

[21]  P. Hall.  Large Sample Optimality of Least Squares Cross-Validation in Density Estimation , 1983 .

[22]  E. F. Schuster,et al.  On the Nonconsistency of Maximum Likelihood Nonparametric Density Estimators , 1981 .

[23]  P. Billingsley,et al.  Probability and Measure , 1980 .

[24]  M. Stone.  An Asymptotic Equivalence of Choice of Model by Cross‐Validation and Akaike's Criterion , 1977 .

[25]  M. Stone.  Asymptotics for and against cross-validation , 1977 .

[26]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[27]  R. Jennrich.  Asymptotic Properties of Non-Linear Least Squares Estimators , 1969 .

[28]  M. Stone.  Application of a Measure of Information to the Design and Comparison of Regression Experiments , 1959 .

[29]  S.,et al.  CONSISTENT CROSS-VALIDATED DENSITY ESTIMATION , 2022 .