A priori power analysis in longitudinal three-level multilevel models: An example with therapist effects

Abstract Over the last few years, three-level longitudinal models have become more common in psychotherapy research, particularly in therapist-effect or group-effect studies. Thus far, limited attention has been paid to power analysis in these models. This article demonstrates the effects of intraclass correlation, level of randomization, sample size, covariates and drop-out on power, using data from a routine outcome monitoring study. Results indicate that randomization at the patient level is the most efficient, and that increasing the number of measurements does not increase power much. Adding a covariate or having a 25% drop-out rate had limited effects on study power in our data. In addition, the results demonstrate that sufficient power can be reached with small sample sizes, but that larger sample sizes are needed to prevent estimation bias for the model parameters and standard errors.

[1]  W. Lutz,et al.  Assessing treatment progress of individual patients using expected treatment response models. , 2001, Journal of consulting and clinical psychology.

[2]  L. Hedges,et al.  Intraclass Correlation Values for Planning Group-Randomized Trials in Education , 2007 .

[3]  B. Wampold,et al.  The consequence of ignoring a nested factor on measures of effect size in analysis of variance. , 2000, Psychological methods.

[4]  K. Carroll,et al.  Now you see it, now you don't: a comparison of traditional versus random-effects regression models in the analysis of longitudinal follow-up data from a clinical trial. , 1997, Journal of consulting and clinical psychology.

[5]  Mirjam Moerbeek,et al.  Powerful and Cost-Efficient Designs for Longitudinal Intervention Studies With Two Treatment Groups , 2008 .

[6]  Risto Lethonen.  Multilevel Statistical Models (3rd ed.) , 2005 .

[7]  H. Goldstein.  Multilevel Statistical Models , 2006 .

[8]  G. Tasca,et al.  The relationship between attachment scales and group therapy alliance growth differs by treatment type for women with binge-eating disorder. , 2007 .

[9]  J. Hox,et al.  Sufficient Sample Sizes for Multilevel Modeling , 2005 .

[10]  E. Hawkins,et al.  The therapeutic effects of providing patient progress information to therapists and patients , 2004 .

[11]  G. A. Marcoulides,et al.  Multilevel Analysis Techniques and Applications , 2002 .

[12]  S. L. Nielsen,et al.  Improving the effects of psychotherapy: The use of early identification of treatment and problem-solving strategies in routine practice. , 2003 .

[13]  Roel Bosker,et al.  PINT (Power IN Two-level designs) , 1999 .

[14]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[15]  D. Bolt,et al.  Therapist effects in psychotherapy: A random-effects modeling of the National Institute of Mental Health Treatment of Depression Collaborative Research Program data , 2006 .

[16]  S. Soldz.  Models and meanings: Therapist effects and the stories we tell , 2006 .

[17]  W. Lutz,et al.  Patient profiling: an application of random coefficient regression models to depicting the response of a patient to outpatient psychotherapy. , 1999, Journal of consulting and clinical psychology.

[18]  W. Lutz,et al.  Prediction of dose-response relations based on patient characteristics. , 2001, Journal of clinical psychology.

[19]  R. Bosker.  Boekbespreking van "A.S. Bryk & S.W. Raudenbusch - Hierarchical linear models: Applications and data analysis methods" : Sage Publications, Newbury Parki, London/New Delhi 1992 , 1995 .

[20]  Sandra Eldridge,et al.  Patterns of intra-cluster correlation from primary care research to inform study design and analysis. , 2004, Journal of clinical epidemiology.

[21]  E. Rafaeli,et al.  Adaptive Modeling of Progress in Outpatient Psychotherapy , 2002 .

[22]  S. Raudenbush.  Educational Applications of Hierarchical Linear Models: A Review , 1988 .

[23]  Naihua Duan,et al.  Multilevel Modeling : Methodological Advances, Issues, and Applications , 2003 .

[24]  P. Spinhoven,et al.  Cognitive-behavioural intervention for self-harm: randomised controlled trial , 2008, British Journal of Psychiatry.

[25]  Mirjam Moerbeek,et al.  Power and money in cluster randomized trials: when is it worth measuring a covariate? , 2006, Statistics in medicine.

[26]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[27]  E. Hawkins,et al.  Improving outcomes for poorly responding clients: the use of clinical support tools and feedback to clients. , 2005, Journal of clinical psychology.

[28]  John S. Lyons,et al.  Therapist Effects in Outpatient Psychotherapy: A Three-Level Growth Curve Approach , 2007 .

[29]  P. Garety,et al.  Cognitive‐Behavioural Interventions , 2011 .

[30]  M. Moerbeek.  Randomization of Clusters Versus Randomization of Persons Within Clusters , 2005 .

[31]  S. L. Nielsen,et al.  The Effects of Providing Therapists With Feedback on Patient Progress During Psychotherapy: Are Outcomes Enhanced? , 2001, Psychotherapy research : journal of the Society for Psychotherapy Research.

[32]  Rien van der Leeden,et al.  Multilevel Analysis of Repeated Measures Data , 1998 .

[33]  S. Raudenbush.  Statistical analysis and optimal design for cluster randomized trials , 1997 .

[34]  R. Hill,et al.  Do early responders to psychotherapy maintain treatment gains? , 2002, Journal of clinical psychology.

[35]  Timothy J. Robinson,et al.  Multilevel Analysis: Techniques and Applications , 2002 .

[36]  M K Campbell,et al.  Cluster trials in implementation research: estimation of intracluster correlation coefficients and sample size. , 2001, Statistics in medicine.

[37]  B. Schaalje,et al.  Psychotherapy quality control: the statistical generation of expected recovery curves for integration into an early warning system , 2001 .

[38]  Roel Bosker,et al.  Standard Errors and Sample Sizes for Two-Level Research , 1993 .

[39]  R. Bosker,et al.  Estimating standard errors of regression coefficients in hierarchical linear models for power calculations USER ' S MANUAL Version 2 . 1 , 2003 .

[40]  Mirjam Moerbeek,et al.  The Consequence of Ignoring a Level of Nesting in Multilevel Analysis , 2004, Multivariate behavioral research.

[41]  Donald Hedeker,et al.  Longitudinal Data Analysis , 2006 .

[42]  I. Elkin,et al.  Therapist effects in the National Institute of Mental Health Treatment of Depression Collaborative Research Program , 2006 .

[43]  R. Gallop,et al.  Therapist effects in the National Institute of Mental Health Treatment of Depression Collaborative Research Program and other psychotherapy studies , 2006 .

[44]  W. Lutz.  Patient-Focused Psychotherapy Research and Individual Treatment Progress as Scientific Groundwork for an Empirically Based Clinical Practice , 2002 .

[45]  Daniel M. Bolt,et al.  Therapist effects: Clever ways to make them (and everything else) disappear , 2006 .

[46]  Steven Teerenstra,et al.  Pseudo cluster randomization: a treatment allocation method to minimize contamination and selection bias , 2005, Statistics in medicine.

[47]  S. L. Nielsen,et al.  Waiting for supershrink: an empirical analysis of therapist effects , 2003 .

[48]  P. Spinhoven,et al.  Predictors of response to the ‘Coping with Depression’ course for older adults. A field study , 2006, Aging & mental health.

[49]  A. Beck,et al.  Meta‐Analysis of Therapist Effects in Psychotherapy Outcome Studies , 1991 .

[50]  S D Imber,et al.  National Institute of Mental Health Treatment of Depression Collaborative Research Program. General effectiveness of treatments. , 1989, Archives of general psychiatry.

[51]  Allan Donner,et al.  Design and Analysis of Cluster Randomization Trials in Health Research , 2001 .

[52]  Roel Bosker,et al.  Multilevel analysis : an introduction to basic and advanced multilevel modeling , 1999 .

[53]  M. J. Bishop,et al.  Comparison of empirically-derived and rationally-derived methods for identifying patients at risk for treatment failure , 2002 .

[54]  Roel Bosker,et al.  PINT. Estimating standard errors of regression coefficients in hierarchical linear models for power calculations. User's manual. , 1999 .