Q-learning: a data analysis method for constructing adaptive interventions.

Increasing interest in individualizing and adapting intervention services over time has led to the development of adaptive interventions. Adaptive interventions operationalize the individualization of a sequence of intervention options over time via the use of decision rules that input participant information and output intervention recommendations. We introduce Q-learning, which is a generalization of regression analysis to settings in which a sequence of decisions regarding intervention options or services is made. The use of Q is to indicate that this method is used to assess the relative quality of the intervention options. In particular, we use Q-learning with linear regression to estimate the optimal (i.e., most effective) sequence of decision rules. We illustrate how Q-learning can be used with data from sequential multiple assignment randomized trials (SMARTs; Murphy, 2005) to inform the construction of a more deeply tailored sequence of decision rules than those embedded in the SMART design. We also discuss the advantages of Q-learning compared to other data analysis approaches. Finally, we use the Adaptive Interventions for Children With ADHD SMART study (Center for Children and Families, University at Buffalo, State University of New York, William E. Pelham as principal investigator) for illustration.

[1]  B. Muthén,et al.  Adaptive designs for randomized trials in public health. , 2009, Annual review of public health.

[2]  Ben J. A. Kröse,et al.  Learning from delayed rewards , 1995, Robotics Auton. Syst..

[3]  Stephen P. Hinshaw,et al.  Behavioral versus Behavioral and Pharmacological Treatment in ADHD Children Attending a Summer Treatment Program , 2000, Journal of abnormal child psychology.

[4]  D. Rivera,et al.  Using engineering control principles to inform the design of adaptive interventions: a conceptual introduction. , 2007, Drug and alcohol dependence.

[5]  William E. Pelham,et al.  Psychosocial and combined treatments for ADHD , 1999 .

[6]  J. Robins.  Addendum to “a new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect” , 1987 .

[7]  J M Robins,et al.  Marginal Mean Models for Dynamic Regimes , 2001, Journal of the American Statistical Association.

[8]  S. Raudenbush,et al.  STUDYING THE CAUSAL EFFECTS OF INSTRUCTION WITH APPLICATION TO PRIMARY-SCHOOL MATHEMATICS , 2003 .

[9]  W. Pelham,et al.  Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorders. , 1992, Journal of the American Academy of Child and Adolescent Psychiatry.

[10]  David J. Radosevich,et al.  The moderating role of goal commitment on the goal difficulty–performance relationship: A meta-analytic review and critical reanalysis. , 1998 .

[11]  P. Thall,et al.  Covariate‐adjusted adaptive randomization in a sarcoma trial with multi‐stage treatments , 2005, Statistics in medicine.

[12]  Joel Greenhouse,et al.  Effects of methylphenidate and expectancy on children with ADHD: behavior, academic performance, and attributions in a summer treatment program and regular classroom settings. , 2002, Journal of consulting and clinical psychology.

[13]  James M. Robins,et al.  Optimal Structural Nested Models for Optimal Sequential Decisions , 2004 .

[14]  Linda Taylor,et al.  On Understanding Intervention in Psychology and Education , 1994 .

[15]  Yitzhak Fried,et al.  Enriching Goal-Setting Theory with Time: An Integrated Approach , 2004 .

[16]  E. Schaughency,et al.  Building Capacity to Implement and Sustain Effective Practices to Better Serve Children , 2006 .

[17]  J L Schafer,et al.  Multiple Imputation for Multivariate Missing-Data Problems: A Data Analyst's Perspective. , 1998, Multivariate behavioral research.

[18]  D. Mackinnon,et al.  A Simulation Study of Mediated Effect Measures. , 1995, Multivariate behavioral research.

[19]  James M. Robins,et al.  Association, Causation, And Marginal Structural Models , 1999, Synthese.

[20]  S. Murphy,et al.  The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. , 2007, American journal of preventive medicine.

[21]  H. Sung,et al.  Selecting Therapeutic Strategies Based on Efficacy and Death in Multicourse Clinical Trials , 2002 .

[22]  William E. Pelham,et al.  Evidence-Based Psychosocial Treatments for Attention-Deficit/Hyperactivity Disorder , 2008, Journal of clinical child and adolescent psychology : the official journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53.

[23]  Roderick J. A. Little,et al.  Statistical Analysis with Missing Data , 1988 .

[24]  S A Murphy,et al.  Screening Experiments for Developing Dynamic Treatment Regimes , 2009, Journal of the American Statistical Association.

[25]  James R McKay,et al.  Adaptive Interventions in Drug Court , 2008, Criminal justice review.

[26]  Susan A Murphy,et al.  Examining clinical judgment in an adaptive intervention design: The fast track program. , 2006, Journal of consulting and clinical psychology.

[27]  N. Kanas.  The theory and practice of group psychotherapy. , 2011, International journal of group psychotherapy.

[28]  Susan Murphy,et al.  Inference for non-regular parameters in optimal dynamic treatment regimes , 2010, Statistical methods in medical research.

[29]  Mark J van der Laan,et al.  Statistical Learning of Origin-Specific Statically Optimal Individualized Treatment Rules , 2007, The international journal of biostatistics.

[30]  A. Tsiatis,et al.  Optimal Estimator for the Survival Distribution and Related Quantities for Treatment Policies in Two‐Stage Randomization Designs in Clinical Trials , 2004, Biometrics.

[31]  T. Dishion,et al.  An adaptive approach to family intervention: linking engagement in family-centered intervention to reductions in adolescent problem behavior. , 2007, Journal of consulting and clinical psychology.

[32]  Edwin A. Locke,et al.  Relationships Among Goal Difficulty, Business Strategies, and Performance On A Complex Management Simulation Task , 1991 .

[33]  Erica E M Moodie,et al.  A Comparison of Variable Selection Approaches for Dynamic Treatment Regimes , 2010, The international journal of biostatistics.

[34]  Biao Zhang,et al.  Empirical Likelihood in Missing Data Problems , 2009 .

[35]  D F Klein,et al.  COGNITIVE THERAPY , 2016 .

[36]  A. Bandura.  Social Foundations of Thought and Action: A Social Cognitive Theory , 1985 .

[37]  Stuart E. Dreyfus,et al.  Applied Dynamic Programming , 1965 .

[38]  Bruce E. Wampold,et al.  Principles of Empirically Supported Interventions in Counseling Psychology , 2002 .

[39]  S. Murphy,et al.  An experimental design for the development of adaptive treatment strategies , 2005, Statistics in medicine.

[40]  S. Pliszka,et al.  Practice parameter for the assessment and treatment of children and adolescents with attention-deficit/hyperactivity disorder. , 2007, Journal of the American Academy of Child and Adolescent Psychiatry.

[41]  Joel B. Greenhouse,et al.  Effects of methylphenidate and expectancy on children with ADHD: behavior, academic performance, and attributions in a summer treatment program and regular classroom settings. , 2002, Journal of Consulting and Clinical Psychology.

[42]  S. Murphy,et al.  Variable Selection for Qualitative Interactions. , 2011, Statistical methodology.

[43]  Marilyn B. Cole.  Group Dynamics in Occupational Therapy: The Theoretical Basis and Practice Application of Group Treatment , 1993 .

[44]  D. Rubin.  [On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.] Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies , 1990 .

[45]  Roger A. Sugden,et al.  Multiple Imputation for Nonresponse in Surveys , 1988 .

[46]  John R. Weisz,et al.  Treatment dissemination and evidence-based practice: Strengthening intervention through clinician-researcher collaboration , 2006 .

[47]  S. Murphy,et al.  Experimental design and primary data analysis methods for comparing adaptive interventions. , 2012, Psychological methods.

[48]  Abbie Brown,et al.  Design experiments: Theoretical and methodological challenges in creating complex interventions in c , 1992 .

[49]  Abraham Wandersman,et al.  Community interventions and effective prevention. , 2003, The American psychologist.

[50]  D. A. Kenny,et al.  The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. , 1986, Journal of personality and social psychology.

[51]  James R McKay,et al.  Is there a case for extended interventions for alcohol and drug use disorders? , 2005, Addiction.

[52]  Mark J van der Laan,et al.  Individualized treatment rules: Generating candidate clinical trials , 2007, Statistics in medicine.

[53]  S. Murphy,et al.  Developing adaptive treatment strategies in substance abuse research. , 2007, Drug and alcohol dependence.

[54]  Chris Watkins,et al.  Learning from delayed rewards , 1989 .

[55]  J. Robins.  A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect , 1986 .

[56]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[57]  J. Webster,et al.  EFFECTS OF FEEDBACK AND COGNITIVE PLAYFULNESS ON PERFORMANCE IN MICROCOMPUTER SOFTWARE TRAINING , 2006 .

[58]  Wayne F Velicer,et al.  Tailored communications for smoking cessation: past successes and future directions. , 2006, Drug and alcohol review.

[59]  Donald B. Rubin,et al.  Bayesian Inference for Causal Effects: The Role of Randomization , 1978 .

[60]  Marie Davidian,et al.  Estimation of Survival Distributions of Treatment Policies in Two‐Stage Randomization Designs in Clinical Trials , 2002, Biometrics.

[61]  Susan A Murphy,et al.  Customizing treatment to the patient: adaptive treatment strategies. , 2007, Drug and alcohol dependence.

[62]  Benjamin B. Lahey,et al.  A Practical Measure of Impairment: Psychometric Properties of the Impairment Rating Scale in Samples of Children With Attention Deficit Hyperactivity Disorder and Two School-Based Samples , 2006, Journal of clinical child and adolescent psychology : the official journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53.