Akaike-type criteria and the reliability of inference: Model selection versus statistical model specification

Since the 1990s, the Akaike Information Criterion (AIC) and its various modifications/extensions, including BIC, have found wide applicability in econometrics as objective procedures that can be used to select parsimonious statistical models. The aim of this paper is to argue that these model selection procedures invariably give rise to unreliable inferences, primarily because their choice within a prespecified family of models (a) assumes away the problem of model validation, and (b) ignores the relevant error probabilities. This paper argues for a return to the original statistical model specification problem, as envisaged by Fisher (1922), where the task is understood as one of selecting a statistical model in such a way as to render the particular data a truly typical realization of the stochastic process specified by the model in question. The key to addressing this problem is to replace trading goodness-of-fit against parsimony with statistical adequacy as the sole criterion for when a fitted model accounts for the regularities in the data.

[1]  R. F.,et al.  Mathematical Statistics , 1944, Nature.

[2]  J. Stoer,et al.  Introduction to Numerical Analysis , 2002 .

[3]  R. Fisher,et al.  On the Mathematical Foundations of Theoretical Statistics , 1922 .

[4]  B. M. Pötscher,et al.  MODEL SELECTION AND INFERENCE: FACTS AND FICTION , 2005, Econometric Theory.

[5]  Deborah G. Mayo,et al.  Methodology in Practice: Statistical Misspecification Testing , 2004, Philosophy of Science.

[6]  I. A. Kieseppä.  Akaike Information Criterion, Curve-fitting, and the Philosophical Problem of Simplicity , 1997, The British Journal for the Philosophy of Science.

[7]  J. Doob.  Stochastic processes , 1953 .

[8]  F. Diebold.  Econometrics: Retrospect and prospect , 2001 .

[9]  Carl Friedrich Gauss.  Theoria motus corporum coelestium , 1981 .

[10]  E. Lehmann.  Model Specification: The Views of Fisher and Neyman, and Later Developments , 1990 .

[11]  R. A. Fisher,et al.  Design of Experiments , 1936 .

[12]  Ian Hacking.  Logic of Statistical Inference , 1965 .

[13]  Aris Spanos,et al.  On theory testing in econometrics : Modeling with nonexperimental data , 1995 .

[14]  W. Greene,et al.  计量经济分析 = Econometric analysis , 2009 .

[15]  Nandini Kannan,et al.  Statistics: Reflections on the Past and Visions for the Future , 2001 .

[16]  A. Spanos.  Revisiting data mining: ‘hunting’ with or without a license , 2000 .

[17]  A. Spanos.  Statistical Misspecification and the Reliability of Inference: The Simple T-Test in the Presence of Markov Dependence , 2009 .

[18]  Aris Spanos,et al.  The simultaneous-equations model revisited: Statistical adequacy and identification , 1990 .

[19]  R. L. Dekock.  Some Comments , 2021 .

[20]  C. L. Mallows.  Some comments on C_p , 1973 .

[21]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[22]  Aris Spanos,et al.  Revisiting the omitted variables argument: Substantive vs. statistical adequacy , 2006 .

[23]  Karl Pearson,et al.  THE FUNDAMENTAL PROBLEM OF PRACTICAL STATISTICS , 1920 .

[24]  Helmut Ltkepohl,et al.  New Introduction to Multiple Time Series Analysis , 2007 .

[25]  R. Fisher,et al.  STATISTICAL METHODS AND SCIENTIFIC INDUCTION , 1955 .

[26]  A. Spanos.  Error and Inference: Theory Testing in Economics and the Error-Statistical Perspective , 2009 .

[27]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[28]  G. Kitagawa,et al.  Information Criteria and Statistical Modeling , 2007 .

[29]  M. Powell,et al.  Approximation theory and methods , 1984 .

[30]  Deborah G. Mayo,et al.  Error and Inference: Recent Exchanges on Experimental Reasoning, Reliability, and the Objectivity and Rationality of Science , 2009 .

[31]  B. Skyrms.  Choice and chance : an introduction to inductive logic , 1968 .

[32]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[33]  A. Spanos.  Where do statistical models come from? Revisiting the problem of specification , 2006, math/0610849.

[34]  Siddharth Kalla.  of Error (Statistics) , 2010 .

[35]  D. Mayo,et al.  Severe Testing as a Basic Concept in a Neyman–Pearson Philosophy of Induction , 2006, The British Journal for the Philosophy of Science.

[36]  Qi Li,et al.  Nonparametric Econometrics: Theory and Practice , 2006 .

[37]  Peter E. Kennedy.  A Guide to Econometrics , 1979 .

[38]  Walter Zucchini,et al.  Model Selection , 2011, International Encyclopedia of Statistical Science.

[39]  Aris Spanos,et al.  Probability theory and statistical inference: econometric modelling with observational data , 1999 .

[40]  D. Cox,et al.  Objectivity and conditionality in frequentist inference , 2010 .

[41]  Jerzy Neyman,et al.  Note on an Article by Sir Ronald Fisher , 1956 .

[42]  A. Spanos.  Philosophy of Econometrics , 2021, The Routledge Handbook of Philosophy of Economics.

[43]  Aris Spanos,et al.  Statistical Foundations of Econometric Modelling , 1986 .

[44]  B. G. Quinn,et al.  The determination of the order of an autoregression , 1979 .

[45]  G. Watson.  Approximation theory and numerical methods , 1980 .

[46]  Malcolm R. Forster,et al.  How to Tell When Simpler, More Unified, or Less Ad Hoc Theories will Provide More Accurate Predictions , 1994, The British Journal for the Philosophy of Science.

[47]  H. Akaike.  Statistical predictor identification , 1970 .

[48]  Geert Dhaene,et al.  Probability Theory and Statistical Inference: Econometric Modeling With Observational Data , 2001 .

[49]  E. Cheney.  Introduction to approximation theory , 1966 .

[50]  A. McQuarrie,et al.  Regression and Time Series Model Selection , 1998 .

[51]  Nils Lid Hjort,et al.  Model Selection and Model Averaging , 2001 .

[52]  D. Cox,et al.  Frequentist statistics as a theory of inductive inference , 2006, math/0610846.

[53]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[54]  Aris Spanos,et al.  Curve Fitting, the Reliability of Inductive Inference, and the Error‐Statistical Approach , 2007, Philosophy of Science.

[55]  D. M. Allen.  Mean Square Error of Prediction as a Criterion for Selecting Variables , 1971 .

[56]  T. J. Rivlin.  An Introduction to the Approximation of Functions , 2003 .

[57]  A. Spanos,et al.  Statistical Adequacy and the Testing of Trend Versus Difference Stationarity , 2003 .

[58]  R. Tweney.  Error and the growth of experimental knowledge , 1998 .

[59]  F. B. Hildebrand,et al.  Introduction To Numerical Analysis , 1957 .

[60]  Aris Spanos,et al.  The Model Specification Problem from a Probabilistic Reduction Perspective , 2001 .

[61]  D. Luenberger.  Optimization by Vector Space Methods , 1968 .

[62]  C. Mallows.  Some Comments on Cp , 2000, Technometrics.