Therapist variation within randomised trials of psychotherapy: implications for precision, internal and external validity

Nesting of patients within therapists in psychotherapy trials creates an additional level within the design. The multilevel nature of this design has implications for the precision, internal and external validity of estimates of the treatment effect. Prior to or during a trial, psychotherapies are allocated to therapists and therapists are assigned to patients such that the therapist becomes part of the causal pathway from the intervention to the patient. It is therefore important to consider not only the relationship between interventions and patients but also relationships between interventions and therapists and between therapists and patients. Research designs comparing the effects of therapeutic approaches, therapist characteristics and packages of the two can be unified by viewing therapists as an important source of variability within psychotherapy outcome studies. Methodological considerations arising from therapist variation will be discussed, drawing together and building upon the associated psychotherapy and statistical literatures. Parallels will also be made with related designs and methods of analysis.

[1]  S. Thompson,et al.  Clustering by health professional in individually randomised trials , 2005, BMJ : British Medical Journal.

[2]  I. Elkin,et al.  Therapist effects in the National Institute of Mental Health Treatment of Depression Collaborative Research Program , 2006 .

[3]  Zac E. Imel,et al.  Untangling the alliance-outcome correlation: exploring the relative importance of therapist and patient variability in the alliance. , 2007, Journal of consulting and clinical psychology.

[4]  Gordon H Guyatt,et al.  Need for expertise based randomised controlled trials , 2005, BMJ : British Medical Journal.

[5]  P. Pilkonis,et al.  Conceptual and methodological issues in comparative studies of psychotherapy and pharmacotherapy, II: Nature and timing of treatment effects. , 1988, The American journal of psychiatry.

[6]  C D Naylor,et al.  Meta-analysis of controlled clinical trials. , 1989, The Journal of rheumatology.

[7]  R. Rapee,et al.  Behavioral treatment of generalized anxiety disorder , 1992 .

[8]  R. Gallop,et al.  Therapist effects in the National Institute of Mental Health Treatment of Depression Collaborative Research Program and other psychotherapy studies , 2006 .

[9]  A T McLellan,et al.  Therapist success and its determinants. , 1985, Archives of general psychiatry.

[10]  Daniel M. Bolt,et al.  Therapist effects: Clever ways to make them (and everything else) disappear , 2006 .

[11]  A Donner,et al.  Statistical considerations in the design and analysis of community intervention trials. , 1996, Journal of clinical epidemiology.

[12]  A. Turvey,et al.  Cognitive therapy vs behaviour therapy in the treatment of chronic general anxiety. , 1987, Behaviour research and therapy.

[13]  R. Wiggins,et al.  Nurse therapist trainee variability: the implications for selection and training. , 2006, Journal of Advanced Nursing.

[14]  Xin Tu,et al.  Therapists as fixed versus random effects-some statistical and conceptual issues: a comment on Siemer and Joormann (2003). , 2003, Psychological methods.

[15]  G. Staines.  Comparative outcome evaluations of psychotherapies: Guidelines for addressing eight limitations of the gold standard of causal inference. , 2007, Psychotherapy.

[16]  B. J. Winer.  Statistical Principles in Experimental Design , 1992 .

[17]  P. Schnurr,et al.  Design of Department of Veterans Affairs Cooperative Study no. 420: group treatment of posttraumatic stress disorder. , 2001, Controlled clinical trials.

[18]  W. Wilkins.  Therapy—therapist confounds in psychotherapy research , 1986, Cognitive Therapy and Research.

[19]  P. Pilkonis,et al.  Do therapists vary much in their success? Findings from four outcome studies. , 1986, The American journal of orthopsychiatry.

[20]  R. Kirk.  Experimental Design: Procedures for the Behavioral Sciences , 1970 .

[21]  A. Mathews,et al.  The effects of relaxation training with cognitive or nondirective therapy and the role of relaxation-induced anxiety in the treatment of generalized anxiety. , 1987, Journal of consulting and clinical psychology.

[22]  Weng Kee Wong,et al.  Sample size formulae for trials comparing group and individual treatments in a multilevel model , 2008, Statistics in medicine.

[23]  Robert S. Kenyon,et al.  United Kingdom Alcohol Treatment Trial (UKATT): hypotheses, design and methods. , 2001, Alcohol and alcoholism.

[24]  B. Wampold,et al.  The consequence of ignoring a nested factor on measures of effect size in analysis of variance. , 2000, Psychological methods.

[25]  H. Goldstein.  Multilevel Statistical Models , 2006 .

[26]  S D Imber,et al.  National Institute of Mental Health Treatment of Depression Collaborative Research Program. General effectiveness of treatments. , 1989, Archives of general psychiatry.

[27]  F. E. Satterthwaite.  An approximate distribution of estimates of variance components. , 1946, Biometrics.

[28]  B. Moser,et al.  The two-sample t test versus satterthwaite's approximate f test , 1989 .

[29]  Allan Donner,et al.  Design and Analysis of Cluster Randomization Trials in Health Research , 2001 .

[30]  A. Mathews,et al.  Generalised anxiety: a controlled treatment study. , 1987, Behaviour research and therapy.

[31]  Jutta Joormann,et al.  Power and measures of effect size in analysis of variance with fixed versus random nested factors. , 2003, Psychological methods.

[32]  M. Fennell,et al.  Comparison of behavior therapy and cognitive behavior therapy in the treatment of generalized anxiety disorder. , 1991, Journal of consulting and clinical psychology.

[33]  Amita K. Manatunga,et al.  Sample Size Estimation in Cluster Randomized Studies with Varying Cluster Size , 2001 .

[34]  Judith Cohen,et al.  A treatment outcome study for sexually abused preschool children: initial findings. , 1996, Journal of the American Academy of Child and Adolescent Psychiatry.

[35]  Peter Fayers,et al.  Statistical evaluation of learning curve effects in surgical trials , 2004, Clinical trials.

[36]  Loretta S. Malta,et al.  A controlled evaluation of cognitive behavioural therapy for posttraumatic stress in motor vehicle accident survivors. , 2003, Behaviour research and therapy.

[37]  A. Kazdin.  Comparative outcome studies of psychotherapy: methodological issues and strategies. , 1986, Journal of consulting and clinical psychology.

[38]  C. Roberts,et al.  The implications of variation in outcome between health professionals for the design and analysis of randomized controlled trials. , 1999, Statistics in medicine.

[39]  Harvey Goldstein,et al.  Multiple membership multiple classification (MMMC) models , 2001 .

[40]  Satterthwaite Fe.  An approximate distribution of estimates of variance components. , 1946 .

[41]  D. Moher,et al.  Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: explanation and elaboration. , 2008, Annals of internal medicine.

[42]  O S Miettinen,et al.  The clinical trial as a paradigm for epidemiologic research. , 1989, Journal of clinical epidemiology.

[43]  H. Melander,et al.  The Subject-by-Formulation Interaction as a Criterion of Interchangeability of Drugs , 1989 .

[44]  Tim Croudace,et al.  Analysis of cluster randomized cross‐over trial data: a comparison of methods , 2007, Statistics in medicine.

[45]  D. Kiesler.  Some myths of psychotherapy research and the search for a paradigm. , 1966 .

[46]  P. Schnurr,et al.  Cognitive behavioral therapy for posttraumatic stress disorder in women: a randomized controlled trial. , 2007, JAMA.

[47]  K. Lovell,et al.  Telephone Administered Cognitive Behaviour Therapy for Treatment of Obsessive Compulsive Disorder: Randomised Controlled Non-inferiority Trial , 2007 .

[48]  S. L. Nielsen,et al.  Waiting for supershrink: an empirical analysis of therapist effects , 2003 .

[49]  J. Mintz,et al.  Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. , 1991, Journal of consulting and clinical psychology.

[50]  M. Lambert.  The individual therapist's contribution to psychotherapy process and outcome , 1989 .

[51]  Andrew Thomas,et al.  WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility , 2000, Stat. Comput..

[52]  Harvey Goldstein,et al.  Multilevel Models in the Study of Dynamic Household Structures , 2000 .

[53]  Jutta Joormann,et al.  Assumptions and consequences of treating providers in therapy studies as fixed versus random effects: reply to Crits-Christoph, Tu, and Gallop (2003) and Serlin, Wampold, and Levin (2003). , 2003, Psychological methods.

[54]  G. Fenton,et al.  Cognitive Therapy, Analytic Psychotherapy and Anxiety Management Training for Generalised Anxiety Disorder , 1994, British Journal of Psychiatry.

[55]  Zac E. Imel,et al.  Psychiatrist effects in the psychopharmacological treatment of depression. , 2006, Journal of affective disorders.

[56]  A Donner,et al.  Sample size calculation for cluster randomized cross‐over trials , 2008, Statistics in medicine.

[57]  John S. Lyons,et al.  Therapist Effects in Outpatient Psychotherapy: A Three-Level Growth Curve Approach , 2007 .

[58]  K. Muller,et al.  A COMPARISON OF POWER APPROXIMATIONS FOR SATTERTHWAITE'S TEST. , 1995, Communications in statistics: Simulation and computation.

[59]  G. L. Paul,et al.  Resurrection of uniformity assumption myths and the fallacy of statistical absolutes in psychotherapy research. , 1978, Journal of consulting and clinical psychology.

[60]  D. Eggett,et al.  An analysis of therapist treatment effects: toward providing feedback to individual therapists on their clients' psychotherapy outcome. , 2006, Journal of clinical psychology.

[61]  S G Thompson,et al.  The design and analysis of paired cluster randomized trials: an application of meta-analysis techniques. , 1997, Statistics in medicine.

[62]  Elizabeth E. Hill,et al.  Cognitive trauma therapy for battered women with PTSD (CTT-BW). , 2004, Journal of consulting and clinical psychology.

[63]  A. Huitson,et al.  Applied Statistics: Analysis of Variance and Regression , 1976 .

[64]  S M Kerry,et al.  Unequal cluster sizes for trials in English and Welsh general practice: implications for sample size calculations. , 2001, Statistics in medicine.

[65]  C. Martindale,et al.  The therapist-as-fixed-effect fallacy in psychotherapy research. , 1978, Journal of consulting and clinical psychology.

[66]  Simon G Thompson,et al.  The use of random effects models to allow for clustering in individually randomized trials , 2005, Clinical trials.

[67]  B. Wampold,et al.  Estimating variability in outcomes attributable to therapists: a naturalistic study of outcomes in managed care. , 2005, Journal of consulting and clinical psychology.

[68]  E. Jones,et al.  Working with feelings: The importance of emotion in both cognitive-behavioral and interpersonal therapy in the NIMH Treatment of Depression Collaborative Research Program. , 2002 .

[69]  P. Schnurr,et al.  Issues in the design of multisite clinical trials of psychotherapy: VA Cooperative Study No. 494 as an example. , 2005, Contemporary clinical trials.

[70]  J. Levin,et al.  Should providers of treatment be regarded as a random factor? If it ain't broke, don't "fix" it: a comment on Siemer and Joormann (2003). , 2003, Psychological methods.

[71]  M. Strack,et al.  Therapist effects on outcome and alliance in inpatient psychotherapy. , 2008, Journal of Clinical Psychology.

[72]  Chris Roberts,et al.  Design and analysis of clinical trials with clustering effects due to treatment , 2005, Clinical trials.

[73]  B. Leese,et al.  Randomised controlled trial comparing cost effectiveness of general practitioners and nurse practitioners in primary care , 2000, BMJ : British Medical Journal.

[74]  David J Torgerson,et al.  Contamination in trials: is cluster randomisation the answer? , 2001, BMJ : British Medical Journal.

[75]  A. Donner,et al.  Pitfalls of and controversies in cluster randomization trials. , 2004, American journal of public health.

[76]  D. Bolt,et al.  Therapist effects in psychotherapy: A random-effects modeling of the National Institute of Mental Health Treatment of Depression Collaborative Research Program data , 2006 .

[77]  S. Soldz.  Models and meanings: Therapist effects and the stories we tell , 2006 .

[78]  P. Pilkonis,et al.  Conceptual and methodological issues in comparative studies of psychotherapy and pharmacotherapy, I: Active ingredients and mechanisms of change. , 1988, The American journal of psychiatry.

[79]  I. Elkin,et al.  Rejoinder to commentaries by Stephen Soldz and Paul Crits-Christoph on therapist effects , 2006 .

[80]  D. Hoover.  Clinical trials of behavioural interventions with heterogeneous teaching subgroup effects , 2002, Statistics in medicine.

[81]  D. Ricks.  Supershrink: Methods of a therapist judged successful on the basis of adult outcomes of adolescent patients. , 1974 .

[82]  I. Elkin,et al.  A Major Dilemma in Psychotherapy Outcome Research: Disentangling Therapists From Therapies , 1999 .

[83]  I. Elkin,et al.  NIMH Treatment of Depression Collaborative Research Program. Background and research plan. , 1985, Archives of general psychiatry.

[84]  H. Samson,et al.  [The personality of the psychotherapist]. , 1950, Laval medical.

[85]  G. Woody,et al.  Is the Counselor an “Active Ingredient” in Substance Abuse Rehabilitation?: An Examination of Treatment Success among Four Counselors , 1988, The Journal of nervous and mental disease.

[86]  A. Horvath.  The Great Psychotherapy Debate: Models Methods and Findings , 2002 .

[87]  G. Woody,et al.  The Psychotherapist Matters: Comparison of Outcomes Across Twenty-Two Therapists and Seven Patient Samples , 1997 .

[88]  D. Bolt,et al.  The consequences of “anchoring” in longitudinal multilevel models: bias in the estimation of patient variability and therapist effects , 2007 .