Prediction Versus Accommodation and the Risk of Overfitting

When a scientist uses an observation to formulate a theory, it is no surprise that the resulting theory accurately captures that observation. However, when the theory makes a novel prediction—when it predicts an observation that was not used in its formulation—this seems to provide more substantial confirmation of the theory. This paper presents a new approach to the vexed problem of understanding the epistemic difference between prediction and accommodation. In fact, there are several problems that need to be disentangled; in all of them, the key is the concept of overfitting. We float the hypothesis that accommodation is a defective methodology only when the methods used to accommodate the data fail to guard against the risk of overfitting. We connect our analysis with the proposals that other philosophers have made. We also discuss its bearing on the conflict between instrumentalism and scientific realism. 1. Introduction2. Predictivisms—a taxonomy3. Observations4. Formulating the problem5. What might Annie be doing wrong?6. Solutions7. Observations explained8. Mayo on severe tests9. The miracle argument and scientific realism10. Concluding comments Introduction Predictivisms—a taxonomy Observations Formulating the problem What might Annie be doing wrong? Solutions Observations explained Mayo on severe tests The miracle argument and scientific realism Concluding comments

[1]  Deborah G. Mayo,et al.  Error and the Growth of Experimental Knowledge , 1996 .

[2]  Colin Howson.  Fitting your theory to the facts: probably not such a bad thing after all , 1990 .

[3]  M. Stone,et al.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[4]  M. Forster,et al.  Key Concepts in Model Selection: Performance and Generalizability. , 2000, Journal of mathematical psychology.

[5]  V. Vieland,et al.  Statistical Evidence: A Likelihood Paradigm , 1998 .

[6]  Carl G. Hempel,et al.  I.—STUDIES IN THE LOGIC OF CONFIRMATION (II.) , 1945 .

[7]  Marc Lange.  The Apparent Superiority of Prediction to Accommodation as a Side Effect: a Reply to Maher , 2001, The British Journal for the Philosophy of Science.

[8]  John Worrall,et al.  Scientific Discovery and Theory-Confirmation , 1985 .

[9]  I. Lakatos.  Changes in the Problem of Inductive Logic , 1968 .

[10]  David R. Anderson,et al.  Model selection and inference : a practical information-theoretic approach , 2000 .

[11]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[12]  Friedrich Stadler.  The Vienna Circle and Logical Empiricism , 2003 .

[13]  H. Feigl,et al.  Minnesota studies in the philosophy of science , 1956 .

[14]  Colin Howson.  Accommodation, Prediction and Bayesian Confirmation Theory , 1988, PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association.

[15]  Malcolm R. Forster,et al.  Predictive Accuracy as an Achievable Goal of Science , 2002, Philosophy of Science.

[16]  Malcolm R. Forster,et al.  How to Tell When Simpler, More Unified, or Less Ad Hoc Theories will Provide More Accurate Predictions , 1994, The British Journal for the Philosophy of Science.

[17]  K. Popper,et al.  Conjectures and refutations;: The growth of scientific knowledge , 1972 .

[18]  R. Tweney.  Error and the growth of experimental knowledge , 1998 .

[19]  H. Putnam.  Mathematics, Matter, And Method , 1975 .

[20]  John Earman,et al.  Einstein's Explanation of the Motion of Mercury's Perihelion , 1993 .

[21]  A. Musgrave,et al.  Logical versus Historical Theories of Confirmation , 1974, The British Journal for the Philosophy of Science.

[22]  A. McQuarrie,et al.  Regression and Time Series Model Selection , 1998 .

[23]  M. Forster,et al.  Model Selection in Science: The Problem of Language Variance , 1999, The British Journal for the Philosophy of Science.

[24]  N. Goodman.  Fact, Fiction, and Forecast , 1955 .

[25]  M. Stone.  An Asymptotic Equivalence of Choice of Model by Cross‐Validation and Akaike's Criterion , 1977 .

[26]  P. Maher,et al.  Prediction, Accommodation, and the Logic of Discovery , 1988, PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association.

[27]  Deborah G. Mayo,et al.  Novel Evidence and Severe Tests , 1991, Philosophy of Science.

[28]  Elliott Sober,et al.  Two Uses of Unification , 2003 .

[29]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[30]  Nancy J. Nersessian,et al.  The Rise of the Wave Theory of Light: Optical Theory and Experiment in the Early Nineteenth Century , 1991 .

[31]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[32]  Karl R. Popper.  The Logic of Scientific Discovery. , 1977 .

[33]  P. Maher,et al.  How Prediction Enhances Confirmation , 1990 .

[34]  E. Sober.  Instrumentalism, Parsimony, and the Akaike Framework , 2002, Philosophy of Science.

[35]  I. Lakatos,et al.  The problem of inductive logic , 1970 .