Evaluating the Effectiveness of Personalized Medicine With Software

We present methodological advances in understanding the effectiveness of personalized medicine models and supply easy-to-use open-source software. Personalized medicine involves the systematic use of individual patient characteristics to determine which treatment option is most likely to result in a better average outcome for the patient. Why is personalized medicine not done more in practice? One of many reasons is because practitioners do not have any easy way to holistically evaluate whether their personalization procedure does better than the standard of care, termed improvement. Our software, “Personalized Treatment Evaluator” (the R package PTE), provides inference for improvement out-of-sample in many clinical scenarios. We also extend current methodology by allowing evaluation of improvement in the case where the endpoint is binary or survival. In the software, the practitioner inputs 1) data from a single-stage randomized trial with one continuous, incidence or survival endpoint and 2) an educated guess of a functional form of a model for the endpoint constructed from domain knowledge. The bootstrap is then employed on data unseen during model fitting to provide confidence intervals for the improvement for the average future patient (assuming future patients are similar to the patients in the trial). One may also test against a null scenario where the hypothesized personalization are not more useful than a standard of care. We demonstrate our method’s promise on simulated data as well as on data from a randomized comparative trial investigating two treatments for depression.

[1]  R. DeRubeis,et al.  Treatment Selection in Depression. , 2018, Annual review of clinical psychology.

[2]  Donglin Zeng,et al.  New Statistical Learning Methods for Estimating Optimal Dynamic Treatment Regimes , 2015, Journal of the American Statistical Association.

[3]  Michael Berk,et al.  The new field of ‘precision psychiatry’ , 2017, BMC Medicine.

[4]  Thomas A Louis,et al.  Individualized treatment effects with censored data via fully nonparametric Bayesian accelerated failure time models. , 2017, Biostatistics.

[5]  Erica E M Moodie,et al.  Estimating Optimal Dynamic Regimes: Correcting Bias under the Null , 2009, Scandinavian journal of statistics, theory and applications.

[6]  J. Matthews,et al.  Randomization in Clinical Trials: Theory and Practice; , 2003 .

[7]  Zachary D. Cohen,et al.  The Personalized Advantage Index: Translating Research on Prediction into Individualized Treatment Recommendations. A Demonstration , 2014, PloS one.

[8]  Nathan Kallus,et al.  Recursive Partitioning for Personalization using Observational Data , 2016, ICML.

[9]  A. Danese,et al.  Implementing Precision Psychiatry: A Systematic Review of Individualized Prediction Models for Clinical Practice , 2020, Schizophrenia bulletin.

[10]  G. L. Paul Strategy of outcome research in psychotherapy. , 1967, Journal of consulting psychology.

[11]  Susan A. Murphy,et al.  A Generalization Error for Q-Learning , 2005, J. Mach. Learn. Res..

[12]  Marc Ratkovic,et al.  Estimating treatment effect heterogeneity in randomized program evaluation , 2013, 1305.5682.

[13]  Mark J van der Laan,et al.  Targeted Learning of the Mean Outcome under an Optimal Dynamic Treatment Rule , 2015, Journal of causal inference.

[14]  Holly Janes,et al.  Combining biomarkers to optimize patient treatment recommendations , 2014, Biometrics.

[15]  R. DeRubeis,et al.  Prediction of response to medication and cognitive therapy in the treatment of moderate to severe depression. , 2009, Journal of consulting and clinical psychology.

[16]  Leroy Hood,et al.  Systems biology, proteomics, and the future of health care: toward predictive, preventative, and personalized medicine. , 2004, Journal of proteome research.

[17]  I. Mechelen,et al.  Quint: An R package for the identification of subgroups of clients who differ in which treatment alternative is best for them , 2015, Behavior Research Methods.

[18]  R. Dawes Judgment under uncertainty: The robust beauty of improper linear models in decision making , 1979 .

[19]  B. Rothbaum,et al.  Differential responses to psychotherapy versus pharmacotherapy in patients with chronic forms of major depression and childhood trauma , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[20]  J. van Eys,et al.  Interaction between prognostic factors and treatment. , 1983, Controlled clinical trials.

[21]  T. Cai,et al.  Identifying predictive markers for personalized treatment selection , 2016, Biometrics.

[22]  R. DeRubeis,et al.  Cognitive therapy vs medications in the treatment of moderate to severe depression. , 2005, Archives of general psychiatry.

[23]  A. Buja,et al.  Valid post-selection inference , 2013, 1306.1059.

[24]  J. Robins,et al.  The International Journal of Biostatistics CAUSAL INFERENCE Dynamic Regime Marginal Structural Mean Models for Estimation of Optimal Dynamic Treatment Regimes , Part I : Main Content , 2011 .

[25]  Hansheng Wang,et al.  Subgroup Analysis via Recursive Partitioning , 2009, J. Mach. Learn. Res..

[26]  Wenbin Lu,et al.  Variable selection for optimal treatment decision , 2013, Statistical methods in medical research.

[27]  David C. Mohr,et al.  Baseline Depression Severity as Moderator of Depression Outcomes Between Cognitive Behavioral Therapy vs Pharmacotherapy: An Individual Patient Data Meta-analysis. , 2015, JAMA psychiatry.

[28]  D. Byar,et al.  Selecting optimal treatment in clinical trials using covariate information. , 1977, Journal of chronic diseases.

[29]  D. Rubin Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[30]  Erica Moodie,et al.  Dynamic treatment regimes. , 2004, Clinical trials.

[31]  Francesco C Stingo,et al.  Bayesian personalized treatment selection strategies that integrate predictive with prognostic determinants , 2019, Biometrical journal. Biometrische Zeitschrift.

[32]  Susan A. Murphy,et al.  A Conceptual Framework for Adaptive Preventive Interventions , 2004, Prevention Science.

[33]  Lacey Gunter,et al.  Variable Selection for Qualitative Interactions in Personalized Medicine While Controlling the Family-Wise Error Rate , 2011, Journal of biopharmaceutical statistics.

[34]  R. Lalonde Evaluating the Econometric Evaluations of Training Programs with Experimental Data , 1984 .

[35]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[36]  J. Rice Mathematical Statistics and Data Analysis , 1988 .

[37]  D. Wolfe,et al.  Test for qualitative interaction of clinical significance. , 1997, Statistics in medicine.

[38]  A. Agresti An introduction to categorical data analysis , 1997 .

[39]  Yakovlev AYu,et al.  The choice of cancer treatment based on covariate information. , 1994, Statistics in medicine.

[40]  Z. Segal,et al.  Personality and Differential Treatment Response in Major Depression: A Randomized Controlled Trial Comparing Cognitive-Behavioural Therapy and Pharmacotherapy , 2008, Canadian journal of psychiatry. Revue canadienne de psychiatrie.

[41]  Dennis L. Sun,et al.  Exact post-selection inference, with application to the lasso , 2013, 1311.6238.

[42]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[43]  J. Berger,et al.  A Bayesian Approach to Subgroup Identification , 2014, Journal of biopharmaceutical statistics.

[44]  I NICOLETTI,et al.  The Planning of Experiments , 1936, Rivista di clinica pediatrica.

[45]  L. Gunter,et al.  A Simple Method for Variable Selection in Regression with Respect to Treatment Selection , 2011 .

[46]  Julian J. Faraway,et al.  Does data splitting improve prediction? , 2013, Stat. Comput..

[47]  S. Murphy,et al.  An experimental design for the development of adaptive treatment strategies , 2005, Statistics in medicine.

[48]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[49]  D. Cox A note on data-splitting for the evaluation of significance levels , 1975 .

[50]  Erica E M Moodie,et al.  Demystifying Optimal Dynamic Treatment Regimes , 2007, Biometrics.

[51]  Anastasios A. Tsiatis,et al.  Q- and A-learning Methods for Estimating Optimal Dynamic Treatment Regimes , 2012, Statistical science : a review journal of the Institute of Mathematical Statistics.

[52]  M. Gail,et al.  Testing for qualitative interactions between treatment effects and patient subsets. , 1985, Biometrics.

[53]  J. Robins,et al.  Estimation and extrapolation of optimal treatment and testing strategies , 2008, Statistics in medicine.

[54]  I. van Mechelen,et al.  Qualitative interaction trees: a tool to identify qualitative treatment–subgroup interactions , 2014, Statistics in medicine.

[55]  Andreas Buja,et al.  Using recursive partitioning to find and estimate heterogenous treatment effects in randomized clinical trials , 2018, Journal of Experimental Criminology.

[56]  D. Zeng,et al.  Recent development on statistical methods for personalized medicine discovery , 2013, Frontiers of Medicine.

[57]  A. Franco,et al.  Toward a neuroimaging treatment selection biomarker for major depressive disorder. , 2013, JAMA psychiatry.

[58]  D P Byar,et al.  Assessing apparent treatment--covariate interactions in randomized clinical trials. , 1985, Statistics in medicine.

[59]  James M. Robins,et al.  Optimal Structural Nested Models for Optimal Sequential Decisions , 2004 .

[60]  Jason Brinkley,et al.  A Generalized Estimator of the Attributable Benefit of an Optimal Treatment Regime , 2010, Biometrics.

[61]  M. Silvapulle,et al.  Tests Against Qualitative Interaction: Exact Critical Values and Robust Tests , 2001, Biometrics.

[62]  K. Davies The $1,000 Genome: The Revolution in DNA Sequencing and the New Era of Personalized Medicine , 2010 .

[63]  G. Andersson,et al.  PERSONALIZED TREATMENT OF ADULT DEPRESSION: MEDICATION, PSYCHOTHERAPY, OR BOTH? A SYSTEMATIC REVIEW , 2012, Depression and anxiety.

[64]  A. Buja,et al.  Covariance Adjustments for the Analysis of Randomized Field Experiments , 2013, Evaluation review.

[65]  B. Chakraborty,et al.  Statistical Methods for Dynamic Treatment Regimes: Reinforcement Learning, Causal Inference, and Personalized Medicine , 2013 .

[66]  Susan Murphy,et al.  Inference for non-regular parameters in optimal dynamic treatment regimes , 2010, Statistical methods in medical research.

[67]  T. Shakespeare,et al.  Observational Studies , 2003 .

[68]  Eric B. Laber,et al.  A Robust Method for Estimating Optimal Treatment Regimes , 2012, Biometrics.

[69]  D. Freedman,et al.  Weighting Regressions by Propensity Scores , 2008, Evaluation review.

[70]  Daniel J. Lizotte,et al.  Set‐valued dynamic treatment regimes for competing outcomes , 2012, Biometrics.

[71]  L. Hood,et al.  Predictive, personalized, preventive, participatory (P4) cancer medicine , 2011, Nature Reviews Clinical Oncology.

[72]  M. J. van der Laan,et al.  Statistical Issues and Limitations in Personalized Medicine Research with Clinical Trials , 2012, The international journal of biostatistics.

[73]  J. Foster Subgroup Identification and Variable Selection from Randomized Clinical Trial Data. , 2013 .

[74]  Andreas Buja,et al.  Misspecified Mean Function Regression , 2014 .

[75]  Jessica K. Barrett,et al.  Doubly Robust Estimation of Optimal Dynamic Treatment Regimes , 2014, Statistics in biosciences.

[76]  Eric B. Laber,et al.  Inference about the expected performance of a data-driven dynamic treatment regime , 2014, Clinical trials.

[77]  George E. P. Box,et al.  Empirical Model‐Building and Response Surfaces , 1988 .

[78]  S. Murphy,et al.  PERFORMANCE GUARANTEES FOR INDIVIDUALIZED TREATMENT RULES. , 2011, Annals of statistics.

[79]  S. Burger An Introduction to the Study of Experimental Medicine , 1950, The Pharos of Alpha Omega Alpha-Honor Medical Society. Alpha Omega Alpha.

[80]  M. Kosorok,et al.  Q-LEARNING WITH CENSORED DATA. , 2012, Annals of statistics.

[81]  Phil Ansell,et al.  Regret‐Regression for Optimal Dynamic Treatment Regimes , 2010, Biometrics.

[82]  A. Buja,et al.  Calibration for Simultaneity : ( Re ) Sampling Methods for Simultaneous Inference with Applications to Function Estimation and Functional Data , 2022 .

[83]  M. Relling,et al.  Moving towards individualized medicine with pharmacogenomics , 2004, Nature.

[84]  Min Zhang,et al.  Estimating optimal treatment regimes from a classification perspective , 2012, Stat.

[85]  Ian W McKeague,et al.  Estimation of treatment policies based on functional predictors. , 2014, Statistica Sinica.

[86]  Jian-Hua Shao,et al.  Bootstrap Sample Size in Nonregular Cases , 1994 .

[87]  Craig A. Rolling,et al.  Model selection for estimating treatment effects , 2014 .

[88]  S. Murphy,et al.  Optimal dynamic treatment regimes , 2003 .

[89]  Marie Davidian,et al.  Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions. , 2013, Biometrika.

[90]  Adam Kapelner,et al.  A matching procedure for sequential experiments that iteratively learns which covariates improve power. , 2021, Biometrics.