Statistical analysis of group-administered intervention data: Reanalysis of two randomized trials

Abstract Group-administered interventions often create statistical dependencies, which, if ignored, increase the rate of Type I errors. The authors analyzed data from two randomized trials involving group interventions to document the impact of statistical dependency on tests of intervention effects and to provide estimates of statistical dependency. Intraclass correlations ranged from .02 to .12. Adjusting for dependencies increased p values for the tests of intervention effects. The increase in the p values depended on the magnitude of the statistical dependence and available degrees of freedom. Results suggest that the literature may overstate the efficacy of group interventions and imply that it will be important to study why groups create dependencies. The authors discuss how dependencies impact statistical power and how researchers can address this concern.

[1]  I. Elkin,et al.  Therapist effects in the National Institute of Mental Health Treatment of Depression Collaborative Research Program , 2006 .

[2]  I. Yalom,et al.  The Theory and Practice of Group Psychotherapy , 1971, Mental Health.

[3]  A. Donner,et al.  Randomization by cluster. Sample size requirements and analysis. , 1981, American journal of epidemiology.

[4]  D M Murray,et al.  A Monte Carlo Study of Alternative Responses To Intraclass Correlation in Community Trials , 1996, Evaluation review.

[5]  W. Fals-Stewart,et al.  Analytic complexities associated with group therapy in substance abuse treatment research: problems, recommendations, and future directions. , 2006, Experimental and clinical psychopharmacology.

[6]  David M. Murray,et al.  Methods To Reduce The Impact Of Intraclass Correlation In Group-Randomized Trials , 2003, Evaluation review.

[7]  A. Kazdin,et al.  Power to detect differences between alternative treatments in comparative psychotherapy outcome research. , 1989, Journal of consulting and clinical psychology.

[8]  C. Fairburn,et al.  The Eating Disorder Examination (12th edition). , 1993 .

[9]  G. W. Snedecor.  Statistical Methods , 1964 .

[10]  G. Burlingame,et al.  Cohesion in group psychotherapy. , 2001 .

[11]  C. Honts,et al.  Analysis of Variance Versus Bootstrap Procedures for Analyzing Dependent Observations in Small Group Research , 1994 .

[12]  A Donner,et al.  A regression approach to the analysis of data arising from cluster randomization. , 1985, International journal of epidemiology.

[13]  P. Pasulka,et al.  Binge Eating: Nature, Assessment and Treatment , 1994 .

[14]  M. Hamilton.  A RATING SCALE FOR DEPRESSION , 1960, Journal of neurology, neurosurgery, and psychiatry.

[15]  M. Susser.  The tribulations of trials--intervention in communities. , 1995, American journal of public health.

[16]  M. Leszcz,et al.  The theory and practice of group psychotherapy, 5th ed. , 2005 .

[17]  Chris Roberts,et al.  Design and analysis of clinical trials with clustering effects due to treatment , 2005, Clinical trials.

[18]  B. Wampold,et al.  The consequence of ignoring a nested factor on measures of effect size in analysis of variance. , 2000, Psychological methods.

[19]  J. Seeley,et al.  Predicting time to recovery among depressed adolescents treated in two psychosocial group interventions. , 2006, Journal of consulting and clinical psychology.

[20]  E. Stice,et al.  Dissonance and healthy weight eating disorder prevention programs: a randomized efficacy trial. , 2006, Journal of consulting and clinical psychology.

[21]  K. Lichstein,et al.  Fair tests of clinical trials: A treatment implementation model , 1994 .

[22]  P. Lewinsohn,et al.  A course in coping: A cognitive-behavioral approach to the treatment of adolescent depression. , 1996 .

[23]  P. Greenbaum,et al.  Testing for group membership effects during and after treatment: The example of group therapy for smoking cessation , 2002 .

[24]  T. Dishion,et al.  When interventions harm. Peer groups and problem behavior. , 1999, The American psychologist.

[25]  T. Achenbach.  Manual for the child behavior checklist/4-18 and 1991 profile , 1991 .

[26]  David M. Murray,et al.  Design and Analysis of Group- Randomized Trials , 1998 .

[27]  W. Shadish,et al.  Empirically supported treatments or type I errors? Problems with the analysis of data from group-administered treatments. , 2005, Journal of consulting and clinical psychology.

[28]  D. Bolt,et al.  Therapist effects in psychotherapy: A random-effects modeling of the National Institute of Mental Health Treatment of Depression Collaborative Research Program data , 2006 .

[29]  D. Antonuccio,et al.  A cost-effectiveness analysis of cognitive behavior therapy and fluoxetine (prozac) in the treatment of depression * , 1997 .

[30]  D. Chambless,et al.  Training in and Dissemination of Empirically-Validated Psychological Treatments: Report and Recommendations , 1995 .

[31]  D. A. Kenny,et al.  Consequences of violating the independence assumption in analysis of variance. , 1986 .

[32]  David M. Zucker,et al.  An Analysis of Variance Pitfall: The Fixed Effects Analysis in a Nested Design , 1990 .

[33]  J. Seeley,et al.  An efficacy/effectiveness study of cognitive-behavioral treatment for adolescents with comorbid major depression and conduct disorder. , 2004, Journal of the American Academy of Child and Adolescent Psychiatry.

[34]  D. A. Kenny,et al.  The statistical analysis of data from small groups. , 2002, Journal of personality and social psychology.

[35]  D. A. Kenny,et al.  Data analysis in social psychology. , 1998 .

[36]  Jessica B. Janega,et al.  Design and analysis of group-randomized trials: a review of recent practices. , 2004, American journal of public health.

[37]  E. Stice,et al.  Eating disorder diagnostic scale: additional evidence of reliability and validity. , 2004, Psychological assessment.