Online advertising has historically been approached as user targeting and ad-to-user matching problems within sophisticated optimization algorithms. As the research area and ad tech industry have progressed over the last couple of decades, advertisers have increasingly emphasized the causal effect estimation of their ads (aka incrementality) using controlled experiments (or A/B testing). Even though observational approaches have been derived in marketing science since the 80s including media mix models, the availability of online advertising personalization has enabled the deployment of more rigorous randomized controlled experiments with millions of individuals. These evolutions in marketing science, online advertising, and the ad tech industry have posed incredible challenges for engineers, data scientists, and marketers alike. With low effect percentage differences (or lift) and often sparse conversion rates, the development of incrementality testing platforms at scale suggests tremendous engineering challenges in the measurement precision and detailed implementation. Similarly, the correct interpretation of results addressing a business goal within the marketing science domain requires significant data science and experimentation research expertise. All these challenges on the ongoing evolution of the online advertising industry and the heterogeneity of its sources (social, paid search, native, programmatic, etc). In the current tutorial, we propose a practical, grounded view in the incrementality testing landscape, including: The business need Solutions in the literature Design and choices in the development of incrementality testing platform The testing cycle, case studies, and recommendations to effective results delivery Incrementality testing evolution in the industry We will provide first-hand lessons on developing and operationalizing such a platform in a major combined DSP and ad network; these are based on running tens of experiments for up to two months each over the last couple of years.
[1]
D. Rubin.
Causal Inference Using Potential Outcomes
,
2005
.
[2]
Ram Akella,et al.
Experimental Designs and Estimation for Online Display Advertising Attribution in Marketplaces
,
2016,
Mark. Sci..
[3]
Narayan Bhamidipati,et al.
Incrementality Testing in Programmatic Advertising: Enhanced Precision with Double-Blind Designs
,
2021,
WWW.
[4]
Brett R. Gordon,et al.
A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook
,
2018,
Mark. Sci..
[5]
Steven Tadelis,et al.
Consumer Heterogeneity and Paid Search Effectiveness: A Large Scale Field Experiment
,
2014
.
[6]
Steven Tadelis,et al.
Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment: Paid Search Effectiveness
,
2015
.
[7]
Ayman Farahat,et al.
Econometric analysis and digital marketing: how to measure the effectiveness of an ad
,
2013,
WSDM '13.
[8]
Advertising Incrementality Measurement using Controlled Geo-Experiments: The Universal App Campaign Case Study
,
2020
.
[9]
Ron Kohavi,et al.
Trustworthy Online Controlled Experiments
,
2020
.
[10]
Randall A. Lewis,et al.
Ghost Ads: Improving the Economics of Measuring Online Ad Effectiveness
,
2017
.