How Persuasive Is THEORETICAL NOTES a Good Fit? A Comment on Theory Testing

Quantitative theories with free parameters often gain credence when they closely fit data. This is a mistake. A good fit reveals nothing about the flexibility of the theory (how much it cannot fit), the variability of the data (how firmly the data rule out what the theory cannot fit), or the likelihood of other outcomes (perhaps the theory could have fit any plausible result), and a reader needs all 3 pieces of information to decide how much the fit should increase belief in the theory. The use of good fits as evidence is not supported by philosophers of science nor by the history of psychology; there seem to be no examples of a theory supported mainly by good fits that has led to demonstrable progress. A better way to test a theory with free parameters is to determine how the theory constrains possible outcomes (i.e., what it predicts), assess how firmly actual outcomes agree with those constraints, and determine if plausible alternative outcomes would have been inconsistent with the theory, allowing for the variability of the data.

[1]  James A. Anderson,et al.  A theory for the recognition of items from short memorized lists , 1973 .

[2]  R. Nickerson.  Confirmation Bias: A Ubiquitous Phenomenon in Many Guises , 1998 .

[3]  Y. Takane,et al.  Structures in stimulus identification data. , 1992 .

[4]  O. Mowrer.  On the dual nature of learning—a re-interpretation of "conditioning" and "problem-solving." , 1947 .

[5]  Gordon H. Bower,et al.  Application of a model to paired-associate learning , 1961 .

[6]  R. Feynman.  Surely You''re Joking Mr , 1992 .

[7]  James L. McClelland,et al.  A distributed, developmental model of word recognition and naming. , 1989, Psychological review.

[8]  Geoffrey E. Hinton,et al.  Parallel Models of Associative Memory , 1989 .

[9]  Kottler Mj,et al.  From 48 to 46: cytological technique, preconception, and the counting of human chromosomes. , 1974 .

[10]  F. J. Anscombe,et al.  Topics in the Investigation of Linear Relations Fitted by the Method of Least Squares , 1967 .

[11]  R. Nosofsky,et al.  Combining exemplar-based category representations and connectionist learning rules. , 1992, Journal of experimental psychology. Learning, memory, and cognition.

[12]  Merrill M. Flood,et al.  On Stochastic Learning Theory , 1952 .

[13]  W. E. Hick.  Quarterly Journal of Experimental Psychology , 1948, Nature.

[14]  Clifford M. Hurvich.  Mean Square over Degrees of Freedom: New Perspectives on a Model Selection Treasure , 1997 .

[15]  Peter Urbach,et al.  Scientific Reasoning: The Bayesian Approach , 1989 .

[16]  David E. Rumelhart,et al.  A multicomponent theory of the perception of briefly exposed visual displays , 1970 .

[17]  M. Kendall,et al.  The Logic of Scientific Discovery. , 1959 .

[18]  Leo Maurice Hurvich,et al.  Color vision , 1981 .

[19]  K. Wexler.  A review of John R. Anderson's language, memory, and thought , 1978, Cognition.

[20]  James L. McClelland.  On the time relations of mental processes: An examination of systems of processes in cascade. , 1979 .

[21]  James L. McClelland,et al.  On the control of automatic processes: a parallel distributed processing account of the Stroop effect. , 1990, Psychological review.

[22]  R. Hyman.  Stimulus information as a determinant of reaction time. , 1953, Journal of experimental psychology.

[23]  S. Sternberg,et al.  The meaning of additive reaction-time effects: tests of three alternatives , 1993 .

[24]  John R. Anderson.  Language, Memory, and Thought , 1976 .

[25]  K. Popper,et al.  The Logic of Scientific Discovery , 1960 .

[26]  J. C. Johnston,et al.  Repetition effects in word and pseudoword identification: comment on Salasoo, Shiffrin, and Feustel. , 1985, Journal of experimental psychology. General.

[27]  Peter Urbach,et al.  Francis Bacon's Philosophy of Science , 1989 .

[28]  A. Salasoo,et al.  Building permanent memory codes: codification and repetition effects in word identification. , 1985, Journal of experimental psychology. General.

[29]  H. Akaike.  A new look at the statistical model identification , 1974 .

[30]  G. Bower,et al.  Presolution dimensional shifts in concept identification: A test of the sampling with replacement axiom in all-or-none models ☆ , 1966 .

[31]  Hermann Bondi,et al.  The philosopher for science , 1992, Nature.

[32]  Imre Lakatos,et al.  The Methodology of Scientific Research Programmes , 1978 .

[33]  D. L. Hintzman.  Why are Formal Models Useful in Psychology , 1990 .

[34]  Henry L. Roediger,et al.  Retrieval blocks in episodic and semantic memory. , 1982 .

[35]  F. Bacon,et al.  The New Organon, and Related Writings , 1960 .

[36]  Ewald Hering.  Outlines of a theory of the light sense , 1964 .

[37]  F. Gregory Ashby,et al.  Deriving Exact Predictions From the Cascade Model , 1982 .

[38]  D. Rubin,et al.  The Precise Time Course of Retention , 1999 .

[39]  S. Sternberg.  High-Speed Scanning in Human Memory , 1966, Science.

[40]  M E Bitterman,et al.  Quantitative two-process analysis of avoidance conditioning in goldfish. , 1994, Journal of experimental psychology. Animal behavior processes.

[41]  B. Skinner,et al.  Principles of Behavior , 1944 .

[42]  R. Atkinson,et al.  A comparison of paired-associate learning models having different acquisition and retention axioms , 1964 .

[43]  N. Mackintosh,et al.  Conditioning And Associative Learning , 1983 .

[44]  J. V. Santen,et al.  How many parameters can a model have and still be testable , 1985 .

[45]  Charles E. Collyer,et al.  Comparing strong and weak models by fitting them to computer-generated data , 1985 .

[46]  Amy Wenzel,et al.  One hundred years of forgetting: A quantitative description of retention , 1996 .

[47]  Aaron J. Ihde,et al.  The development of modern chemistry , 1964 .

[48]  C. Howson,et al.  Scientific Reasoning: The Bayesian Approach , 1989 .

[49]  V. Coltheart,et al.  On Rumelhart's model of visual information-processing. , 1972, Canadian journal of psychology.

[50]  V. Tamari.  Surely You're Joking, Mr. Feynman! , 1985 .

[51]  D. Massaro.  Some criticisms of connectionist models of human performance , 1988 .

[52]  W. Estes.  Toward a Statistical Theory of Learning. , 1994 .

[53]  J. Rodgers,et al.  Social contagion and adolescent sexual behavior: a developmental EMOSA model. , 1993, Psychological review.