Assessing model mimicry using the parametric bootstrap

Abstract We present a general sampling procedure to quantify model mimicry, defined as the ability of a model to account for data generated by a competing model. This sampling procedure, called the parametric bootstrap cross-fitting method (PBCM; cf. Williams (J. R. Statist. Soc. B 32 (1970) 350; Biometrics 26 (1970) 23)), generates distributions of differences in goodness-of-fit expected under each of the competing models. In the data informed version of the PBCM, the generating models have specific parameter values obtained by fitting the experimental data under consideration. The data informed difference distributions can be compared to the observed difference in goodness-of-fit to allow a quantification of model adequacy. In the data uninformed version of the PBCM, the generating models have a relatively broad range of parameter values based on prior knowledge. Application of both the data informed and the data uninformed PBCM is illustrated with several examples.

[1]  R. Ratcliff,et al.  A diffusion model analysis of the effects of aging on brightness discrimination , 2003, Perception & psychophysics.

[2]  J. Dunn.  Model complexity: The fit to random data reconsidered , 2000, Psychological research.

[3]  Petar M. Djuric,et al.  Asymptotic MAP criteria for model selection , 1998, IEEE Trans. Signal Process..

[4]  R. Ratcliff.  Group reaction time distributions and an analysis of distribution statistics. , 1979, Psychological bulletin.

[5]  Anthony C. Davison,et al.  Bootstrap Methods and Their Application , 1998 .

[6]  James O. Berger,et al.  Simulation Methods for Model Criticism and Robustness Analysis , 1998 .

[7]  James J. Heckman,et al.  Handbook of Econometrics, Volume 7A , 2001 .

[8]  Roger Ratcliff.  A note on mimicking additive reaction time models , 1988 .

[9]  Mark A. Pitt,et al.  Global Model Analysis by Landscaping , 2003 .

[10]  Genshiro Kitagawa,et al.  Selected papers of Hirotugu Akaike , 1998 .

[11]  Jonathan P. Bollback,et al.  Bayesian model adequacy and choice in phylogenetics. , 2002, Molecular biology and evolution.

[12]  Wayne A. Wickelgren,et al.  Network Strength Theory of Storage and Retrieval Dynamics. , 1976 .

[13]  M. Aitkin.  Posterior Bayes Factors , 1991 .

[14]  Grünwald,et al.  Model Selection Based on Minimum Description Length. , 2000, Journal of mathematical psychology.

[15]  B. Dosher,et al.  Serial Retrieval Processes in the Recovery of Order Information , 1993 .

[16]  Jorma Rissanen,et al.  Fisher information and stochastic complexity , 1996, IEEE Trans. Inf. Theory.

[17]  John Geweke,et al.  Federal Reserve Bank of Minneapolis Research Department Staff Report 249 Using Simulation Methods for Bayesian Econometric Models: Inference, Development, and Communication , 2022 .

[18]  G. McLachlan.  On Bootstrapping the Likelihood Ratio Test Statistic for the Number of Components in a Normal Mixture , 1987 .

[19]  Walter N. Thurman.  Bayesian Specification Analysis in Econometrics: Comment , 2001 .

[20]  S. S. Wilks.  The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses , 1938 .

[21]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[22]  R. Ratcliff,et al.  Estimating parameters of the diffusion model: Approaches to dealing with contaminant reaction times and parameter variability , 2002, Psychonomic bulletin & review.

[23]  Harley Bornbach,et al.  An introduction to mathematical learning theory , 1967 .

[24]  Philip L. Smith,et al.  A comparison of sequential sampling models for two-choice reaction time. , 2004, Psychological review.

[25]  Raul Cano.  On The Bayesian Bootstrap , 1992 .

[26]  D. Kirsh,et al.  Proceedings of the 25th annual conference of the Cognitive Science Society , 2003 .

[27]  I. J. Myung,et al.  Toward a method of selecting among computational models of cognition. , 2002, Psychological review.

[28]  Anthony C. Atkinson,et al.  A Method for Discriminating between Models , 1970 .

[29]  Purushottam W. Laud,et al.  Predictive Model Selection , 1995 .

[30]  C. Lebiere,et al.  The Atomic Components of Thought , 1998 .

[31]  Richard M. Golden.  Making correct statistical inferences using a wrong probability model , 1995 .

[32]  I. J. Myung,et al.  Applying Occam’s razor in modeling cognition: A Bayesian approach , 1997 .

[33]  George E. P. Box,et al.  Sampling and Bayes' inference in scientific modelling and robustness , 1980 .

[34]  D A Williams.  Discrimination between regression models to determine the pattern of enzyme synthesis in synchronous cell cultures. , 1970, Biometrics.

[35]  R. Ratcliff,et al.  A diffusion model analysis of the effects of aging on letter discrimination. , 2003, Psychology and aging.

[36]  R. Ratcliff,et al.  The effects of aging on reaction time in a signal detection task. , 2001, Psychology and aging.

[37]  Roger Ratcliff,et al.  Statistical mimicking of reaction time data: Single-process models, parameter variability, and mixtures , 1995, Psychonomic bulletin & review.

[38]  Felix Wichmann,et al.  The psychometric function: II. Bootstrap-based confidence intervals and sampling , 2001, Perception & psychophysics.

[39]  Barbara Anne Dosher,et al.  The effects of delay and interference: A speed-accuracy study , 1981, Cognitive Psychology.

[40]  R Ratcliff,et al.  Continuous versus discrete information processing modeling accumulation of partial information. , 1988, Psychological review.

[41]  J. MacKinnon,et al.  Bootstrap tests: how many bootstraps? , 2000 .

[42]  Francis Tuerlinckx,et al.  Diagnostic checks for discrete data regression models using posterior predictive simulations , 2000 .

[43]  E. Mammen.  The Bootstrap and Edgeworth Expansion , 1997 .

[44]  J. Townsend.  SOME RESULTS CONCERNING THE IDENTIFIABILITY OF PARALLEL AND SERIAL PROCESSES , 1972 .

[45]  D. Massaro.  Some criticisms of connectionist models of human performance , 1988 .

[46]  Cutting,et al.  Accuracy, Scope, and Flexibility of Models. , 2000, Journal of mathematical psychology.

[47]  A. Raftery.  Bayesian Model Selection in Social Research , 1995 .

[48]  R. Golden.  Discrepancy Risk Model Selection Test theory for comparing possibly misspecified or nonnested models , 2003 .

[49]  W. Estes,et al.  Traps in the route to models of memory and decision , 2002, Psychonomic bulletin & review.

[50]  James L. McClelland,et al.  The time course of perceptual choice: the leaky, competing accumulator model. , 2001, Psychological review.

[51]  H. Pashler.  STEVENS' HANDBOOK OF EXPERIMENTAL PSYCHOLOGY , 2002 .

[52]  George E. Ferris,et al.  An Introduction to Mathematical Learning Theory , 1966 .

[53]  P. J. Green,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[54]  Xiao-Li Meng,et al.  Posterior Predictive $p$-Values , 1994 .

[55]  Iven Van Mechelen,et al.  A Bayesian approach to the selection and testing of mixture models , 2003 .

[56]  D. Andrews,et al.  A Three-Step Method for Choosing the Number of Bootstrap Repetitions , 2000 .

[57]  L. Wasserman,et al.  The Selection of Prior Distributions by Formal Rules , 1996 .

[58]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[59]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[60]  Susan R. Wilson,et al.  Two guidelines for bootstrap hypothesis testing , 1991 .

[61]  D. Massaro,et al.  Integration of featural information in speech perception. , 1978, Psychological review.

[62]  Z. D. Feng,et al.  Using Bootstrap Likelihood Ratio in Finite Mixture Models , 1994 .

[63]  Estes Wk.  The problem of inference from curves based on group data. , 1956 .

[64]  Frederick Mosteller,et al.  Stochastic Models for Learning , 1956 .

[65]  N. Anderson.  Foundations of information integration theory , 1981 .

[66]  E. Wagenmakers,et al.  AIC model selection using Akaike weights , 2004, Psychonomic bulletin & review.

[67]  Robert Tibshirani,et al.  Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy , 1986 .

[68]  Albert T. Corbett,et al.  Priming and Retrieval from Short-Term Memory: A Speed Accuracy Trade-Off Analysis , 1980 .

[69]  Iven Van Mechelen,et al.  A Bayesian approach to the selection and testing of latent class models , 2000 .

[70]  J. Horowitz.  Chapter 52 The Bootstrap , 2001 .

[71]  Simon Farrell,et al.  Naïve Nonparametric Bootstrap Model Weights Are Biased , 2004, Biometrics.

[72]  Q. Vuong.  Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses , 1989 .

[73]  W H Batchelder,et al.  A measurement-theoretic analysis of the fuzzy logic model of perception. , 1995, Psychological review.

[74]  R. Stine,et al.  Bootstrapping Goodness-of-Fit Measures in Structural Equation Models , 1992 .

[75]  James L. McClelland,et al.  The TRACE model of speech perception , 1986, Cognitive Psychology.

[76]  A. Rollett,et al.  The Monte Carlo Method , 2004 .

[77]  L. Wasserman,et al.  Computing Bayes Factors by Combining Simulation and Asymptotic Approximations , 1997 .

[78]  W. Estes.  The problem of inference from curves based on group data. , 1956, Psychological bulletin.

[79]  James L. McClelland,et al.  The Morton-Massaro law of information integration: implications for models of perception. , 2001, Psychological review.

[80]  J. Berger.  Statistical Decision Theory and Bayesian Analysis , 1988 .

[81]  D. Massaro,et al.  Bayes factor of model selection validates FLMP , 2001, Psychonomic bulletin & review.

[82]  Williams Da,et al.  Discrimination between regression models to determine the pattern of enzyme synthesis in synchronous cell cultures. , 1970 .

[83]  R Ratcliff,et al.  The effects of aging on reaction time in a signal detection task. , 2001, Psychology and aging.

[84]  Jay I. Myung,et al.  Assessing the distinguishability of models and the informativeness of data , 2004, Cognitive Psychology.

[85]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[86]  Albert T. Corbett.  Retrieval dynamics for rote and visual image mnemonics , 1977 .

[87]  D. Massaro,et al.  Models of integration given multiple sources of information. , 1990, Psychological review.

[88]  B. Dosher,et al.  Serial position and set size in short-term memory: The time course of recognition , 1989 .

[89]  A. V. Reed,et al.  List length and the time course of recognition in immediate memory , 1976, Memory & cognition.

[90]  Jeffrey N. Rouder,et al.  A diffusion model account of masking in two-choice letter identification. , 2000, Journal of experimental psychology. Human perception and performance.

[91]  R. Ratcliff.  Methods for dealing with reaction time outliers. , 1993, Psychological bulletin.

[92]  D. Massaro.  Perceiving talking faces: from speech perception to a behavioral principle , 1999 .

[93]  John Geweke,et al.  Using Simulation Methods for Bayesian Econometric Models , 1999 .

[94]  D. Rubin.  Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician , 1984 .

[95]  P. Hall.  Theoretical Comparison of Bootstrap Confidence Intervals , 1988 .

[96]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[97]  Woojae Kim,et al.  Flexibility versus generalizability in model selection , 2003, Psychonomic bulletin & review.

[98]  William H. Press,et al.  Numerical recipes , 1990 .

[99]  Wasserman,et al.  Bayesian Model Selection and Model Averaging. , 2000, Journal of mathematical psychology.

[100]  Joseph P. Romano,et al.  A Review of Bootstrap Confidence Intervals , 1988 .

[101]  M. Carrasco,et al.  The temporal dynamics of visual search: evidence for parallel processing in feature and conjunction searches. , 1999, Journal of experimental psychology. Human perception and performance.

[102]  Roger Ratcliff,et al.  A Theory of Memory Retrieval. , 1978 .

[103]  M. Kendall,et al.  Kendall's advanced theory of statistics , 1995 .

[104]  P. Gr,et al.  Model Selection Based on Minimum Description Length , 1999 .

[105]  Wayne A. Wickelgren,et al.  Speed-accuracy tradeoff and information processing dynamics , 1977 .

[106]  Debashis Kushary,et al.  Bootstrap Methods and Their Application , 2000, Technometrics.

[107]  Jeffrey N. Rouder,et al.  Modeling Response Times for Two-Choice Decisions , 1998 .

[108]  Charles E. Collyer,et al.  Comparing strong and weak models by fitting them to computer-generated data , 1985 .

[109]  John Geweke,et al.  Bayesian Specification Analysis in Econometrics , 2001 .

[110]  Jorma Rissanen,et al.  Strong optimality of the normalized ML models as universal codes and information in data , 2001, IEEE Trans. Inf. Theory.

[111]  Roger Ratcliff,et al.  A diffusion model account of the lexical decision task. , 2004, Psychological review.

[112]  T. Zandt.  Analysis of Response Time Distributions , 2002 .

[113]  J. Cutting,et al.  Selectivity, scope, and simplicity of models: a lesson from fitting judgments of perceived depth. , 1992, Journal of experimental psychology. General.

[114]  John Aitchison,et al.  Statistical Prediction Analysis , 1975 .