Counterfactual-based Incrementality Measurement in a Digital Ad-Buying Platform

The problem of measuring the true incremental effectiveness of a digital advertising campaign is of increasing importance to marketers. With a large and increasing percentage of digital advertising delivered via Demand-Side-Platforms (DSPs) executing campaigns via Real-Time-Bidding (RTB) auctions and programmatic approaches, a measurement solution that satisfies both advertiser concerns and the constraints of a DSP is of particular interest. MediaMath (a DSP) has developed the first practical, statistically sound randomization-based methodology for causal ad effectiveness (or Ad Lift) measurement by a DSP (or similar digital advertising execution system that may not have full control over the advertising transaction mechanisms). We describe our solution and establish its soundness within the causal framework of counterfactuals and potential outcomes, and present a Gibbs-sampling procedure for estimating confidence intervals around the estimated Ad Lift. We also address practical complications (unique to the digital advertising setting) that stem from the fact that digital advertising is targeted and measured via identifiers (e.g., cookies, mobile advertising IDs) that may not be stable over time. One such complication is the repeated occurrence of identifiers, leading to interference among observations. Another is due to the possibility of multiple identifiers being associated with the same consumer, leading to "contamination" with some of their identifiers being assigned to the Treatment group and others to the Control group. Complications such as these have severely impaired previous efforts to derive accurate measurements of lift in practice. In contrast to a few other papers on the subject, this paper has an expository aim as well, and provides a rigorous, self-contained, and readily-implementable treatment of all relevant concepts.

[1]  G. Imbens,et al.  Machine Learning Methods for Estimating Heterogeneous Causal Eects , 2015 .

[2]  Ram Akella,et al.  Marketing campaign evaluation in targeted display advertising , 2012, ADKDD '12.

[3]  Susan Athey,et al.  Recursive partitioning for heterogeneous causal effects , 2015, Proceedings of the National Academy of Sciences.

[4]  Uri Shalit,et al.  Learning Representations for Counterfactual Inference , 2016, ICML.

[5]  Philip Resnik,et al.  GIBBS SAMPLING FOR THE UNINITIATED , 2010 .

[6]  Kevin Leyton-Brown,et al.  Counterfactual Prediction with Deep Instrumental Variables Networks , 2016, ArXiv.

[7]  P. Austin.  An introduction to propensity-score methods for reducing confounding in observational studies , 2011 .

[8]  Brett R. Gordon,et al.  A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook , 2018, Mark. Sci..

[9]  Michael Bailey,et al.  People and Cookies: Imperfect Treatment Assignment in Online Experiments , 2016, WWW.

[10]  David Maxwell Chickering,et al.  A Clinician's Tool for Analyzing Non-Compliance , 1996, AAAI/IAAI, Vol. 2.

[11]  D. Rubin,et al.  Bayesian inference for causal effects in randomized experiments with noncompliance , 1997 .

[12]  Rong Ge,et al.  Evaluating online ad campaigns in a pipeline: causal models at scale , 2010, KDD.

[13]  Garrett A. Johnson.  Ghost Ads: Improving the Economics of Measuring Ad Effectiveness , 2015 .

[14]  R J Little,et al.  Causal effects in clinical and epidemiological studies via potential outcomes: concepts and analytical approaches. , 2000, Annual review of public health.

[15]  D. Rubin.  Causal Inference Using Potential Outcomes , 2005 .