Assumptions and consequences of treating providers in therapy studies as fixed versus random effects: reply to Crits-Christoph, Tu, and Gallop (2003) and Serlin, Wampold, and Levin (2003).

In their comments on the authors' article, R. C. Serlin, B. E. Wampold, and J. R. Levin and P. Crits-Christoph, X. Tu, and R. Gallop took issue with the authors' suggestion to evaluate therapy studies with nested providers with a fixed model approach. In this rejoinder, the authors' comment on Serlin et al's critique by showing that their arguments do not apply, are based on misconceptions about the purpose and nature of statistical inference, or are based on flawed reasoning. The authors also comment on Crits-Christoph et al's critique by showing that the proposed approach is very similar to, but less inclusive than, their own suggestion.

[1]  G. Keppel.  Design and analysis: A researcher's handbook, 3rd ed. , 1991 .

[2]  Anthony C. Davison,et al.  Bootstrap Methods and Their Application , 1998 .

[3]  Scott E. Maxwell,et al.  Designing Experiments and Analyzing Data: A Model Comparison Perspective , 1990 .

[4]  David A. Freedman,et al.  A Nonstochastic Interpretation of Reported Significance Levels , 1983 .

[5]  D. Rubin.  [On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.] Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies , 1990 .

[6]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[7]  Jim Mintz,et al.  Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. , 1991 .

[8]  Jutta Joormann,et al.  Power and measures of effect size in analysis of variance with fixed versus random nested factors. , 2003, Psychological methods.

[9]  David Papineau,et al.  The Virtues of Randomization , 1994, The British Journal for the Philosophy of Science.

[10]  B. Wampold,et al.  The consequence of ignoring a nested factor on measures of effect size in analysis of variance. , 2000, Psychological methods.

[11]  C. Lunneborg,et al.  Random assignment of available cases: bootstrap standard errors and confidence intervals. , 2001, Psychological methods.

[12]  B. Manly.  Randomization, Bootstrap and Monte Carlo Methods in Biology , 2018 .

[13]  Peter Urbach,et al.  Scientific Reasoning: The Bayesian Approach , 1989 .

[14]  E. Pitman.  Significance Tests Which May be Applied to Samples from Any Populations , 1937 .

[15]  R. Serlin,et al.  Misuse of statistical test in three decades of psychotherapy research. , 1994, Journal of consulting and clinical psychology.

[16]  C. Lunneborg.  Data Analysis by Resampling: Concepts and Applications , 1999 .

[17]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .

[18]  W. Hays.  Statistics for the social sciences , 1973 .

[19]  G. Keppel,et al.  Design and Analysis: A Researcher's Handbook , 1976 .

[20]  S. Maxwell,et al.  The proof of the pudding: an illustration of the relative strengths of null hypothesis, meta-analysis, and Bayesian analysis. , 2000, Psychological methods.

[21]  J. Levin,et al.  Should providers of treatment be regarded as a random factor? If it ain't broke, don't "fix" it: a comment on Siemer and Joormann (2003). , 2003, Psychological methods.

[22]  Charles S. Reichardt,et al.  Justifying the use and increasing the power of a t test for a randomized experiment with a convenience sample. , 1999 .

[23]  J. S. Hunter,et al.  Statistics for experimenters : an introduction to design, data analysis, and model building , 1979 .

[24]  L. A. Marascuilo,et al.  Statistical Methods for the Social and Behavioral Sciences. , 1989 .

[25]  David Hogben,et al.  Computer science and statistics :: tenth annual symposium on the interface , 1978 .

[26]  T. Speed,et al.  On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9 , 1990 .

[27]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[28]  A. Bohart,et al.  Foundations of Clinical and Counseling Psychology , 1988 .

[29]  Ronald C. Serlin,et al.  Equivalence confidence intervals for two-group comparisons of means , 1998 .

[30]  P. Holland.  Statistics and Causal Inference , 1985 .

[31]  L. Harlow,et al.  What if there were no significance tests , 1997 .

[32]  R. Tibshirani,et al.  An introduction to the bootstrap , 1993 .

[33]  Robert Fildes,et al.  Journal of business and economic statistics 5: Garcia-Ferrer, A. et al., Macroeconomic forecasting using pooled international data, (1987), 53-67 , 1988 .

[34]  E. Edgington,et al.  Randomization Tests (3rd ed.) , 1998 .

[35]  D. Johnstone.  On the Necessity for Random Sampling , 1989, The British Journal for the Philosophy of Science.

[36]  Xin Tu,et al.  Therapists as fixed versus random effects-some statistical and conceptual issues: a comment on Siemer and Joormann (2003). , 2003, Psychological methods.

[37]  B. Wampold.  The Great Psychotherapy Debate: Models, Methods, and Findings , 2001 .