Decision criteria for model comparison using the parametric bootstrap cross-fitting method

When computational cognitive models are compared regarding their ability to fit empirical data, it is important to consider the models' complexity. The parametric bootstrap cross-fitting method (PBCM, Wagenmakers, Ratcliff, Gomez, & Iverson, 2004) is a promising approach to model comparison and selection that takes the compared models' complexity into account. Applying the PBCM requires solving a classification problem, in which it needs to be determined whether a goodness of fit value generated from the compared models is more likely under one or the other of two existing distributions. Previous literature on the PBCM provides little explicit information on (a) the properties of the distributions one should expect to arise in the scope of the PBCM or (b) which methods for solving the classification problem may be suitable (in which situations). This lack of information may hamper use of the PBCM by cognitive modelers. As part of our general endeavor to make sophisticated modeling methods more available and accessible to cognitive scientists developing computational models, in this article we provide detailed analyses of both the distributions that can be expected to arise when employing the PBCM and the performance characteristics of 8 classification methods. Simulation studies involving 6 artificial pairs of distributions and pairs of distributions arising from 8 pairs of existing cognitive models indicate (a) that the relative location but not the shape of the two distributions can be expected to be constrained and (b) that the k-nearest neighbor method constitutes a good general choice for solving the classification problem.