A permutation test for assessing the presence of individual differences in treatment effects

An important goal of personalized medicine is to identify heterogeneity in treatment effects and then use that heterogeneity to target the intervention to those most likely to benefit. Heterogeneity is assessed using the predicted individual treatment effects framework, and a permutation test is proposed to establish if significant heterogeneity is present given the covariates and predictive model or algorithm used for predicted individual treatment effects. We first show evidence for heterogeneity in the effects of treatment across an illustrative example data set. We then use simulations with two different predictive methods (linear regression model and Random Forests) to show that the permutation test has adequate type-I error control. Next, we use an example dataset as the basis for simulations to demonstrate the ability of the permutation test to find heterogeneity in treatment effects for a predicted individual treatment effects estimate as a function of both effect size and sample size. We find that the proposed test has good power for detecting heterogeneity in treatment effects when the heterogeneity was due primarily to a single predictor, or when it was spread across the predictors. Power was found to be greater for predictions from a linear model than from random forests. This non-parametric permutation test can be used to test for significant differences across individuals in predicted individual treatment effects obtained with a given set of covariates using any predictive method with no additional assumptions.

[1]  Guoqiang Peter Zhang,et al.  Neural networks for classification: a survey , 2000, IEEE Trans. Syst. Man Cybern. Part C.

[2]  Uri Shalit,et al.  Estimating individual treatment effect: generalization bounds and algorithms , 2016, ICML.

[3]  M. Case,et al.  Treatment Response Trajectories and Their Antecedents in Recent-Onset Psychosis: A 2-Year Prospective Study , 2010, Journal of clinical psychopharmacology.

[4]  Korepanova Natalia,et al.  Survival Forests under Test: Impact of the Proportional Hazards Assumption on Prognostic and Predictive Forests for ALS Survival , 2019 .

[5]  Jeremy MG Taylor,et al.  Permutation Testing for Treatment–Covariate Interactions and Subgroup Identification , 2016, Statistics in biosciences.

[6]  The clinical course of alcohol use disorders: using joinpoint analysis to aid in interpretation of growth mixture models. , 2013, Drug and alcohol dependence.

[7]  Zhi Geng,et al.  Assessing the Treatment Effect Heterogeneity with a Latent Variable , 2016, 1603.02712.

[8]  Alan Agresti,et al.  Order-restricted inference for monotone trend alternatives in contigency tables , 1998 .

[9]  Keith Zvoch.  Treatment Fidelity in Multisite Evaluation , 2009 .

[10]  I. Lipkovich,et al.  Tutorial in biostatistics: data‐driven subgroup identification and analysis in clinical trials , 2017, Statistics in medicine.

[11]  Daniel L. Oberski,et al.  Identification of predicted individual treatment effects in randomized clinical trials , 2018, Statistical methods in medical research.

[12]  Justin Grimmer,et al.  Estimating Heterogeneous Treatment Effects and the Effects of Heterogeneous Treatments with Ensemble Methods , 2017, Political Analysis.

[13]  Frank E. Harrell,et al.  Prediction models need appropriate internal, internal-external, and external validation. , 2016, Journal of clinical epidemiology.

[14]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[15]  R. Saunders,et al.  Best Care at Lower Cost: The Path to Continuously Learning Health Care in America , 2013 .

[16]  T. Hothorn,et al.  Individual treatment effect prediction for amyotrophic lateral sclerosis patients , 2016, Statistical methods in medical research.

[17]  Richard K. Crump,et al.  Nonparametric Tests for Treatment Effect Heterogeneity , 2006, The Review of Economics and Statistics.

[18]  D. Hedeker,et al.  A RANDOM-EFFECTS MIXTURE MODEL FOR CLASSIFYING TREATMENT RESPONSE IN LONGITUDINAL CLINICAL TRIALS , 2001, Journal of biopharmaceutical statistics.

[19]  P. Rosenbaum,et al.  Combining group-based trajectory modeling and propensity score matching for causal inferences in nonexperimental longitudinal data. , 2008, Developmental psychology.

[20]  Stefan Wager,et al.  Estimation and Inference of Heterogeneous Treatment Effects using Random Forests , 2015, Journal of the American Statistical Association.

[21]  D. Green,et al.  Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees , 2012 .

[22]  Nicolás M Ballarini,et al.  Subgroup identification in clinical trials via the predicted individual treatment effect , 2018, PloS one.

[23]  S. Senn.  Mastering variation: variance components and personalised medicine , 2015, Statistics in medicine.

[24]  Johann S. Hawe,et al.  Crowdsourced analysis of clinical trial data to predict amyotrophic lateral sclerosis progression , 2014, Nature Biotechnology.

[25]  Matt Taddy,et al.  Heterogeneous Treatment Effects in Digital Experimentation , 2014, 1412.8563.

[26]  I. Lipkovich,et al.  Subgroup identification based on differential effect search—A recursive partitioning method for establishing response to treatment in patient subpopulations , 2011, Statistics in medicine.

[27]  B. Muthén,et al.  Estimating drug effects in the presence of placebo response: Causal inference using growth mixture modeling , 2009, Statistics in medicine.

[28]  David Madigan,et al.  Latent Class Mixture Models of Treatment Effect Heterogeneity , 2017 .

[29]  H. Chipman,et al.  BART: Bayesian Additive Regression Trees , 2008, 0806.3286.

[30]  I. van Mechelen,et al.  Qualitative interaction trees: a tool to identify qualitative treatment–subgroup interactions , 2014, Statistics in medicine.

[31]  J. Berger,et al.  A Bayesian Approach to Subgroup Identification , 2014, Journal of biopharmaceutical statistics.

[32]  P. Holland.  Statistics and Causal Inference: Rejoinder , 1986 .

[33]  L. Marangell,et al.  COMPREHENSIVE REVIEW OF FACTORS IMPLICATED IN THE HETEROGENEITY OF RESPONSE IN DEPRESSION , 2012, Depression and anxiety.

[34]  Alex Dmitrienko,et al.  Survey Results on Industry Practices and Challenges in Subgroup Analysis in Clinical Trials , 2015 .

[35]  D. Rubin.  Causal Inference Using Potential Outcomes , 2005 .

[36]  Graham Dunn,et al.  Modelling treatment‐effect heterogeneity in randomized controlled trials of complex interventions (psychological treatments) , 2007, Statistics in medicine.

[37]  D. Hedeker,et al.  Evaluation of Longitudinal Intervention Effects: An Example of Latent Growth Mixture Models for Ordinal Drug-Use Outcomes , 2010 .

[38]  Euan A Ashley,et al.  The precision medicine initiative: a new national effort. , 2015, JAMA.

[39]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[40]  M. Vaughn,et al.  Heterogeneity in Drug Abuse Among Juvenile Offenders , 2013, International journal of offender therapy and comparative criminology.

[41]  W. Loh,et al.  A regression tree approach to identifying subgroups with differential treatment effects , 2014, Statistics in medicine.

[42]  A. Ng,et al.  Clinical Value of Predicting Individual Treatment Effects for Intensive Blood Pressure Therapy: A Machine Learning Experiment to Estimate Treatment Effects from Randomized Trial Data , 2019, Circulation. Cardiovascular quality and outcomes.

[43]  P. Rosenbaum.  Conditional Permutation Tests and the Propensity Score in Observational Studies , 1984 .