The IOS Test for Model Misspecification

A new test of model misspecification is proposed, based on the ratio of in-sample and out-of-sample likelihoods. The test is broadly applicable and, in simple problems, approximates well-known, intuitive methods. Using jackknife influence curve approximations, it is shown that the test statistic can be viewed asymptotically as a multiplicative contrast between two estimates of the information matrix, both of which are consistent under correct model specification. This approximation is used to show that the statistic is asymptotically normally distributed, although it is suggested that p values be computed using the parametric bootstrap. The resulting methodology is demonstrated with various examples and simulations involving both discrete and continuous data.

[1]  Walter Zucchini,et al.  Model Selection , 2011, International Encyclopedia of Statistical Science.

[2]  E. S. Pearson,et al.  Tests for departure from normality: Comparison of powers , 1977 .

[3]  D. Harville.  Matrix Algebra From a Statistician's Perspective , 1998 .

[4]  A. Gardner.  Methods of Statistics , 1941 .

[5]  R. Dennis Cook,et al.  Detection of Influential Observation in Linear Regression , 2000, Technometrics.

[6]  E. L. Lehmann,et al.  Theory of point estimation , 1950 .

[7]  M Zelen,et al.  Estimation of exponential survival probabilities with concomitant information. , 1965, Biometrics.

[8]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[9]  L. Stefanski,et al.  The Calculus of M-Estimation , 2002 .

[10]  R. Fisher.  Statistical Methods for Research Workers , 1971 .

[11]  R. Beran.  Prepivoting Test Statistics: A Bootstrap View of Asymptotic Refinements , 1988 .

[12]  Ross Ihaka,et al.  Gentleman R: R: A language for data analysis and graphics , 1996 .

[13]  A. Agresti.  An introduction to categorical data analysis , 1997 .

[14]  A. Gelfand,et al.  Bayesian Model Choice: Asymptotics and Exact Calculations , 1994 .

[15]  Richard A. Johnson,et al.  Applied Multivariate Statistical Analysis , 1983 .

[16]  J. Horowitz.  Bootstrap-based critical values for the information matrix test , 1994 .

[17]  W W Piegorsch,et al.  Estimation and Testing with Overdispersed Proportions Using the Beta‐Logistic Regression Model of Heckman and Willis , 2000, Biometrics.

[18]  M. Stone.  An Asymptotic Equivalence of Choice of Model by Cross‐Validation and Akaike's Criterion , 1977 .

[19]  C. I. Bliss.  THE CALCULATION OF THE DOSAGE-MORTALITY CURVE , 1935 .

[20]  S. Geisser,et al.  A Predictive Approach to Model Selection , 1979 .

[21]  Power of the kurtosis statistic, b2, in tests of departures from normality , 1973 .

[22]  M. J. Bayarri,et al.  P Values for Composite Null Models , 2000 .

[23]  Predictive discordancy tests for exponential observations , 1989 .

[24]  J. Shao,et al.  The jackknife and bootstrap , 1996 .

[25]  D. Sloane,et al.  An Introduction to Categorical Data Analysis , 1996 .

[26]  R. Cook.  Detection of influential observation in linear regression , 2000 .

[27]  James M. Robins,et al.  Asymptotic Distribution of P Values in Composite Null Models , 2000 .

[28]  D. Boos,et al.  Monte Carlo Evaluation of Resampling-Based Hypothesis Tests , 2000 .

[29]  H. White.  Maximum Likelihood Estimation of Misspecified Models , 1982 .

[30]  Michel Loève,et al.  Probability Theory I , 1977 .

[31]  F. Scholz.  Maximum Likelihood Estimation , 2006 .

[32]  J. Hausman.  Specification tests in econometrics , 1978 .

[33]  H. White.  A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity , 1980 .