Therapist effects: Clever ways to make them (and everything else) disappear

The purpose of Kim, Wampold, and Bolt’s (2005) analysis of the National Institute of Mental Health (NIMH) Treatment of Depression Collaborative Research Program (TDCRP) data was to estimate the proportion of variance of outcome resulting from therapists (i.e., therapist effects) and compare the size of the therapist effect with that resulting from treatment. To this end, they used multilevel modeling and tested various models that used pretest and posttest measures (or last available scores). Kim et al. found that about 8% of the variance in outcomes was due to therapists, whereas 0% was due to treatment, a result consistent with prior analyses of clinical trials (e.g., Crits-Christoph, 1991). Elkin, Falconnier, Martinovich, and Mahoney (2005) also used multilevel models to estimate therapist effects in the NIMH TDCRP data and found these effects to be negligible. Elkin et al. concluded, based on their analysis and scrutiny of previous studies, that therapists are unimportant (i.e., therapist effects are small or nonexistent). The discrepancy may, as Soldz (2006) contends, be explained by choice of models (i.e., various models produce various results). This is unsatisfying: The data are speaking, but enunciation is a problem. On the other hand, it may be that there is a right answer, as CritsChristoph and Gallop (2006) claim. The goal of any statistical analysis is to choose models that adequately represent the phenomenon being investigated. Failure to reject the null hypothesis may well be due to the fact that no true effect exists. However, failure to reject the null hypothesis may be due to the fact that the model chosen makes assumptions about the phenomenon that are unreasonable: The phenomenon has been forced to contort itself into the shape of the model (Serlin, Wampold, & Levin, 2003). As noted by Serlin et al., ‘‘Once the models are in place, it is the purpose of the experiment and inferential statistical procedures to enable the researcher to criticize all aspects of the model*/most importantly, of course, the hypotheses in question, but also the appropriateness of the model and the assumptions underlying the statistical methods’’ (p. 527). We demonstrate that Elkin et al. chose a model and performed various operations that increased the likelihood that therapist effects will be absent.