Power and measures of effect size in analysis of variance with fixed versus random nested factors.

Ignoring a nested factor can influence the validity of statistical decisions about treatment effectiveness. Previous discussions have centered on consequences of ignoring nested factors versus treating them as random factors on Type I errors and measures of effect size (B. E. Wampold & R. C. Serlin). The authors (a) discuss circumstances under which the treatment of nested provider effects as fixed as opposed to random is appropriate; (b) present 2 formulas for the correct estimation of effect sizes when nested factors are fixed; (c) present the results of Monte Carlo simulations of the consequences of treating providers as fixed versus random on effect size estimates, Type I error rates, and power; and (d) discuss implications of mistaken considerations of provider effects for the study of differential treatment effects in psychotherapy research.

[1]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[2]  L Luborsky,et al.  Comparative studies of psychotherapies: is it true that "everybody has won and all must have prizes"? , 1975, Proceedings of the annual meeting of the American Psychopathological Association.

[3]  Scott E. Maxwell,et al.  Designing Experiments and Analyzing Data: A Model Comparison Perspective , 1990 .

[4]  A. Kazdin.  Methodology, design, and evaluation in psychotherapy research. , 1994 .

[5]  James D. Church,et al.  Comments on Clark's “The language-as-fixed-effect fallacy” , 1976 .

[6]  G. Keppel.  Design and analysis: A researcher's handbook, 3rd ed. , 1991 .

[7]  K. McGraw,et al.  Forming inferences about some intraclass correlation coefficients. , 1996 .

[8]  The Analysis of Random Effects in Modeling Studies. , 1979 .

[9]  Frederick A. Stich,et al.  A meta-analysis of outcome studies comparing bona fide psychotherapies: Empiricially, "all must have prizes." , 1997 .

[10]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[11]  Martijn P. F. Berger,et al.  Design Issues for Experiments in Multilevel Populations , 2000 .

[12]  B. Wampold,et al.  The consequence of ignoring a nested factor on measures of effect size in analysis of variance. , 2000, Psychological methods.

[13]  Stephen W. Raudenbush,et al.  Hierarchical linear models and experimental design. , 1993 .

[14]  G. Keppel,et al.  Design and Analysis: A Researcher's Handbook , 1976 .

[15]  Windy Dryden,et al.  Handbook of Psychotherapy and Behavior Change , 1987, Journal of Cognitive Psychotherapy.

[16]  M. Seligman,et al.  The effectiveness of psychotherapy. The Consumer Reports study. , 1995, The American psychologist.

[17]  S. Raudenbush.  Statistical analysis and optimal design for cluster randomized trials , 1997 .

[18]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .

[19]  B. Wampold.  The Great Psychotherapy Debate: Models, Methods, and Findings , 2001 .

[20]  K. Hopkins.  The Unit of Analysis: Group Means Versus Individual Observations , 1982 .

[21]  W. N. Venables,et al.  Estimation of Variance Components and Applications. , 1990 .

[22]  L. K. Edwards,et al.  Applied analysis of variance in behavioral science , 1993 .

[23]  G. Fenton,et al.  Cognitive Therapy, Analytic Psychotherapy and Anxiety Management Training for Generalised Anxiety Disorder , 1994, British Journal of Psychiatry.

[24]  D. Bonge,et al.  The Experimenter-as-Fixed-Effect Fallacy , 1992 .

[25]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[26]  Harold R. Lindman.  Analysis of Variance in Experimental Design , 1991 .

[27]  Charles S. Reichardt,et al.  Justifying the use and increasing the power of a t test for a randomized experiment with a convenience sample. , 1999 .

[28]  Geoffrey Keppel,et al.  Discussion of Wike and Church's comments , 1976 .

[29]  Roel Bosker,et al.  Standard Errors and Sample Sizes for Two-Level Research , 1993 .

[30]  E. B. Coleman,et al.  Generalization effects vs random effects: Is σTL2 a source of type 1 or type 2 error? , 1979 .

[31]  H. H. Clark.  The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. , 1973 .

[32]  B. J. Winer.  Statistical Principles in Experimental Design , 1992 .

[33]  I. Elkin.  The NIMH Treatment of Depression Collaborative Research Program: Where we began and where we are. , 1994 .

[34]  Jim Mintz,et al.  Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. , 1991 .