Predicting Counterfactuals from Large Historical Data and Small Randomized Trials

When a new treatment is considered for use, whether a pharmaceutical drug or a search engine ranking algorithm, a typical question that arises is, will its performance exceed that of the current treatment? The conventional way to answer this counterfactual question is to estimate the effect of the new treatment in comparison to that of the conventional treatment by running a controlled, randomized experiment. While this approach theoretically ensures an unbiased estimator, it suffers from several drawbacks, including the difficulty in finding representative experimental populations as well as the cost of running randomized trials. Moreover, such trials neglect the huge quantities of available control-condition data, which in principle can be utilized for the harder task of predicting individualized effects. In this paper, we propose a discriminative framework for predicting the outcomes of a new treatment from a large dataset of the control condition and data from a small (and possibly unrepresentative) randomized trial comparing new and old treatments. Our learning objective, which requires minimal assumptions on the treatments, models the relation between the outcomes of the different conditions. This allows us to not only estimate mean effects but also to generate individual predictions for examples outside the small randomized sample. We demonstrate the utility of our approach through experiments in three areas: search engine operation, treatments to diabetes patients, and market value estimation of houses. Our results demonstrate that our approach can reduce the number and size of the currently performed randomized controlled experiments, thus saving significant time, money and effort on the part of practitioners.

[1]  Illtyd Trethowan.  Causality , 1938 .

[2]  References , 1971 .

[3]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[4]  J. Robins,et al.  Doubly Robust Estimation in Missing Data and Causal Inference Models , 2005, Biometrics.

[5]  R. Tibshirani,et al.  Sparsity and smoothness via the fused lasso , 2005 .

[6]  H. Chipman,et al.  Bayesian Additive Regression Trees , 2006 .

[7]  M. J. van der Laan,et al.  Causal Effect Models for Realistic Individualized Treatment and Intention to Treat Rules , 2007, The international journal of biostatistics.

[8]  Thomas Lengauer,et al.  Multi-task learning for HIV therapy screening , 2008, ICML '08.

[9]  Ron Kohavi,et al.  Controlled experiments on the web: survey and practical guide , 2009, Data Mining and Knowledge Discovery.

[10]  Kilian Q. Weinberger,et al.  Feature hashing for large scale multitask learning , 2009, ICML '09.

[11]  Koby Crammer,et al.  A theory of learning from different domains , 2010, Machine Learning.

[12]  H. Chipman,et al.  BART: Bayesian Additive Regression Trees , 2008, 0806.3286.

[13]  Wei Chu,et al.  A contextual-bandit approach to personalized news article recommendation , 2010, WWW '10.

[14]  Ron Kohavi,et al.  Trustworthy online controlled experiments: five puzzling outcomes explained , 2012, KDD.

[15]  W. Marsden.  I and J , 2012 .

[16]  D. Torgerson,et al.  Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials | Cabinet Office , 2012 .

[17]  Bernhard Schölkopf,et al.  On causal and anticausal learning , 2012, ICML.

[18]  Joaquin Quiñonero Candela,et al.  Counterfactual reasoning and learning systems: the example of computational advertising , 2013, J. Mach. Learn. Res..

[19]  Beata Strack,et al.  Impact of HbA1c Measurement on Hospital Readmission Rates: Analysis of 70,000 Clinical Database Patient Records , 2014, BioMed research international.

[20]  Lihong Li,et al.  Counterfactual Estimation and Optimization of Click Metrics for Search Engines , 2014, ArXiv.

[21]  Ryen W. White,et al.  Seeking Insights About Cycling Mood Disorders via Anonymized Search Logs , 2014, Journal of medical Internet research.

[22]  Lihong Li,et al.  Counterfactual Estimation and Optimization of Click Metrics in Search Engines: A Case Study , 2015, WWW.

[23]  Thorsten Joachims,et al.  Counterfactual Risk Minimization , 2015, ICML.

[24]  Thorsten Joachims,et al.  The Self-Normalized Estimator for Counterfactual Learning , 2015, NIPS.

[25]  David Page,et al.  Machine Learning for Treatment Assignment: Improving Individualized Risk Attribution , 2015, AMIA.

[26]  Lihong Li,et al.  Toward Predicting the Outcome of an A/B Experiment for Search Relevance , 2015, WSDM.

[27]  Uri Shalit,et al.  Learning Representations for Counterfactual Inference , 2016, ICML.

[28]  Stefan Wager,et al.  Estimation and Inference of Heterogeneous Treatment Effects using Random Forests , 2015, Journal of the American Statistical Association.

[29]  Cornetto.  Test , 2020, ACM Transactions on Algorithms.