How to detect cognitive strategies: commentary on 'Differentiation and integration: guiding principles for analyzing cognitive change'.

We have to warn our readers: This is a commentary about measurement and statistics. As developmental psychologists, we would like to discuss more interesting topics like cognitive strategy construction, learning, and development. The contribution of Siegler and Chen (2008) is clearly exciting in this respect. We fully agree with Siegler and Chen about the importance of differentiation and integration in the process of cognitive change. We also greatly appreciate the results on self-explanations of both correct and incorrect answers. These findings may be of great practical use in education. However, we reluctantly set all this aside, in favor of a commentary about the use of statistical techniques to classify subjects into categories. In the present case, the categories are rules or strategies for solving water displacement items. The task of accurately classifying children in this way is a crucial step in many developmental studies. To this end, Siegler and Chen use the Rule Assessment Methodology (RAM), originally proposed by Siegler (1976, 1981) for proportional reasoning tasks. We contend that this classification procedure is suboptimal, because it lacks a sound statistical basis. We also contend that there is a better method, which can handle the model implied by RAM, and which is well developed both statistically and computationally. Our message is simple. Psychologists are intensive users of statistical tests. In almost every paper, they apply advanced multivariate techniques such as structural equation modeling, analysis of variance, mixed effect and loglinear analysis, and sequential multinomial logit analysis (see Siegler & Chen, 2008). Given the general reliance on statistical procedures in our empirical studies, it is remarkable that, when it concerns classification, we tend to ignore well-developed statistical procedures, and resort to methods based on ad hoc criteria for classification. In classifying subjects into unobserved categories (types, rules, strategies, etc.) based on observed responses, we should use appropriate statistical techniques, such as finite mixture modeling or categorical latent variable techniques. From a theoretical point of view, we view Siegler’s introduction of Rule Assessment Methodology (RAM) in cognitive task analysis as one of the milestones in the study of cognitive development. The combination of psychological theory (here differentiation and integration) and the analysis of patterns of responses to a set of well-chosen items constitutes a major improvement on traditional interviewing methods, which suffer from problems of objectivity, falsifiability, and replicability. However, from a statistical modeling point of view, we judge RAM to be suboptimal. This is not to say that RAM, in the hands of experts like Siegler and Chen, necessarily produces incorrect results. But RAM as a statistical method can be improved greatly by using adequate statistical techniques based on latent structure and finite mixture modeling (Clogg, 1995). Two basic techniques are latent class analysis (LCA) and binomial mixture analysis (BMA). These techniques (a) prevent us from accepting categories that are not actually supported by the data; (b) enable us to detect unanticipated categories; (c) allow us to estimate model parameters (and their standard errors) that characterize and help us to interpret the categories; (d) allow us to compare different classifications using statistical goodness of fit measures; (e) allow us to classify subjects optimally into the established categories (or latent classes); (f ) allow us to generalize our results to the population. This commentary is not the place to explain the details of LCA or BMA. Fortunately, for both many clear introductions exist (Clogg, 1995; Everitt & Hand, 1981; McCutcheon, 1987; Hagenaars & McCutcheon, 2002), and various commercial and free software packages are available. Most popular packages nowadays are Latent Gold, Panmark, and Mplus. For the analysis presented here we used the free R-package poLCA (Linzer & Lewis, 2007). In cognitive developmental psychology, Thomas and colleagues introduced and extensively applied the