A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook

Observational methods often fail to accurately recover the treatment effects generated from randomized advertising experiments on Facebook.

[1]  David H. Reiley,et al.  Here, there, and everywhere: correlated online behaviors can lead to overestimates of the effects of advertising , 2011, WWW.

[2]  D. Rubin,et al.  Constructing a Control Group Using Multivariate Matched Sampling Methods That Incorporate the Propensity Score , 1985 .

[3]  Analyst,et al.  USING SINGLE-SOURCE DATA TO MEASURE ADVERTISING EFFECTIVENESS , 2016 .

[4]  Paul R. Rosenbaum,et al.  Comparison of Multivariate Matching Methods: Structures, Distances, and Algorithms , 1993 .

[5]  J. Wooldridge.  Inverse probability weighted estimation for general missing data problems , 2004 .

[6]  Avi Goldfarb,et al.  Online Display Advertising: Targeting and Obtrusiveness , 2011, Mark. Sci..

[7]  G. W. Imbens.  Sensitivity to Exogeneity Assumptions in Program Evaluation , 2003 .

[8]  Donald B. Rubin,et al.  Estimating the Causal Effects of Marketing Interventions Using Propensity Score Methodology , 2006 .

[9]  G. Imbens,et al.  Matching on the Estimated Propensity Score , 2009 .

[10]  Andrew Gelman,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2006 .

[11]  K. Imai,et al.  Covariate balancing propensity score , 2014 .

[12]  Paul J. Ferraro,et al.  The performance of non-experimental designs in the evaluation of environmental programs: A design-replication study using a large-scale randomized experiment as a benchmark , 2014 .

[13]  J. Angrist,et al.  Identification and Estimation of Local Average Treatment Effects , 1995 .

[14]  Marco Caliendo,et al.  Some Practical Guidance for the Implementation of Propensity Score Matching , 2005, SSRN Electronic Journal.

[15]  Martin Bichler,et al.  Responsible Data Science , 2017, Bus. Inf. Syst. Eng..

[16]  D. Andrews,et al.  A Three-Step Method for Choosing the Number of Bootstrap Repetitions , 2000 .

[17]  A. W. Kemp,et al.  Kendall's Advanced Theory of Statistics. , 1994 .

[18]  Garrett A. Johnson.  Ghost Ads: Improving the Economics of Measuring Ad Effectiveness , 2015 .

[19]  Dean Eckles,et al.  Bias and High-Dimensional Adjustment in Observational Studies of Peer Effects , 2017, ArXiv.

[20]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[21]  Joseph P. Romano,et al.  Large Sample Confidence Regions Based on Subsamples under Minimal Assumptions , 1994 .

[22]  Timothy W. Armistead.  Resurrecting the Third Variable: A Critique of Pearl's Causal Analysis of Simpson's Paradox , 2014 .

[23]  Jeffrey H Silber,et al.  Optimal multivariate matching before randomization. , 2004, Biostatistics.

[24]  D. Rubin.  Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation , 2001, Health Services and Outcomes Research Methodology.

[25]  D. McCaffrey,et al.  Propensity score estimation with boosted regression for evaluating causal effects in observational studies. , 2004, Psychological methods.

[26]  Rajeev Dehejia,et al.  Propensity Score-Matching Methods for Nonexperimental Causal Studies , 2002, Review of Economics and Statistics.

[27]  Leonard M. Lodish,et al.  How T.V. Advertising Works: A Meta-Analysis of 389 Real World Split Cable T.V. Advertising Experiments , 1995 .

[28]  A. Ichino,et al.  From Temporary Help Jobs to Permanent Employment: What Can We Learn from Matching Estimators and Their Sensitivity? , 2006, SSRN Electronic Journal.

[29]  D. Rubin,et al.  Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction , 2016 .

[30]  G. Imbens,et al.  Approximate residual balancing: debiased inference of average treatment effects in high dimensions , 2016, 1604.07125.

[31]  Kirthi Kalyanam,et al.  Cross channel effects of search engine advertising on brick & mortar retail sales: Meta analysis of large scale field experiments on Google.com , 2017 .

[32]  Harikesh S. Nair,et al.  Native Advertising, Sponsorship Disclosure and Consumer Deception: Evidence from Mobile Search-Ad Experiments , 2017 .

[33]  Randall A. Lewis,et al.  When Less is More: Data and Power in Advertising Experiments , 2015 .

[34]  Daniel Westreich,et al.  Propensity score estimation : machine learning and classification methods as alternatives to logistic regression , 2010 .

[35]  Steven Tadelis,et al.  Consumer Heterogeneity and Paid Search Effectiveness: A Large Scale Field Experiment , 2014 .

[36]  D. Rubin,et al.  Assessing Sensitivity to an Unobserved Binary Covariate in an Observational Study with Binary Outcome , 1983 .

[37]  Bart J. Bronnenberg,et al.  Do Digital Video Recorders Influence Sales? , 2010 .

[38]  F. Götze,et al.  RESAMPLING FEWER THAN n OBSERVATIONS: GAINS, LOSSES, AND REMEDIES FOR LOSSES , 2012 .

[39]  Elizabeth A Stuart,et al.  Matching methods for causal inference: A review and a look forward. , 2010, Statistical science : a review journal of the Institute of Mathematical Statistics.

[40]  G. Imbens,et al.  On the Failure of the Bootstrap for Matching Estimators , 2006 .

[41]  Jeffrey M. Wooldridge,et al.  Solutions Manual and Supplementary Materials for Econometric Analysis of Cross Section and Panel Data , 2003 .

[42]  W. G. Cochran.  The effectiveness of adjustment by subclassification in removing bias in observational studies. , 1968, Biometrics.

[43]  Angus Deaton.  Instruments, Randomization, and Learning about Development , 2010 .

[44]  Christopher R. Taber,et al.  Selection on Observed and Unobserved Variables: Assessing the Effectiveness of Catholic Schools , 2000, Journal of Political Economy.

[45]  Brett R. Gordon,et al.  A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook , 2017 .

[46]  Donald B. Rubin,et al.  Matching methods for causal inference: Designing observational studies , 2007 .

[47]  Navdeep S. Sahni.  Effect of temporal spacing between advertising exposures: Evidence from online field experiments , 2015 .

[48]  Antonio F. Galvao,et al.  Bayesian Endogeneity Bias Modeling , 2014 .

[49]  Michaela Draganska,et al.  Internet versus Television Advertising: A Brand-Building Comparison , 2013 .

[50]  T. T. Pham,et al.  A Deep Causal Inference Approach to Measuring the Effects of Forming Group Loans in Online Non-profit Microfinance Platform , 2017, 1706.02795.

[51]  R. Lalonde.  Evaluating the Econometric Evaluations of Training Programs with Experimental Data , 1984 .

[52]  J. Zubizarreta.  Journal of the American Statistical Association Using Mixed Integer Programming for Matching in an Observational Study of Kidney Failure after Surgery Using Mixed Integer Programming for Matching in an Observational Study of Kidney Failure after Surgery , 2022 .

[53]  Bo Lu,et al.  Functions for Optimal Non-Bipartite Matching , 2016 .

[54]  I. Executive.  An Evaluation of Methods Used to Assess the Effectiveness of Advertising on the Internet , 2010 .

[55]  David H. Reiley,et al.  Online ads and offline sales: measuring the effect of retail advertising via a controlled experiment on Yahoo! , 2014 .

[56]  Donald B. Rubin,et al.  Bayesian Inference for Causal Effects: The Role of Randomization , 1978 .

[57]  P. Raghavendra Rau,et al.  Changing Names with Style: Mutual Fund Name Changes and Their Effects on Fund Flows , 2004 .

[58]  Leslie Wood,et al.  CROSS PLATFORM SALES IMPACT: CRACKING THE CODE ON SINGLE SOURCE , 2013 .

[59]  G. Imbens,et al.  Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score , 2000 .

[60]  J. Robins,et al.  Toward a curse of dimensionality appropriate (CODA) asymptotic theory for semi-parametric models. , 1997, Statistics in medicine.

[61]  R. Maitra,et al.  Supplement to “ A k-mean-directions Algorithm for Fast Clustering of Data on the Sphere ” published in the Journal of Computational and Graphical Statistics , 2009 .

[62]  P. Rosenbaum.  Sensitivity analysis for certain permutation inferences in matched observational studies , 1987 .

[63]  Kevin Arceneaux,et al.  A Cautionary Note on the Use of Matching to Estimate Causal Effects: An Empirical Example Comparing Matching Estimates to an Experimental Benchmark , 2010 .

[64]  Justin M. Rao,et al.  The Unfavorable Economics of Measuring the Returns to Advertising , 2014 .

[65]  David H. Reiley,et al.  Location, Location, Location: Repetition and Proximity Increase Advertising Effectiveness , 2016 .

[66]  A. Belloni,et al.  SPARSE MODELS AND METHODS FOR OPTIMAL INSTRUMENTS WITH AN APPLICATION TO EMINENT DOMAIN , 2012 .

[67]  D. Rubin,et al.  Principal Stratification in Causal Inference , 2002, Biometrics.

[68]  Peter E. Rossi.  Invited Paper - Even the Rich Can Make Themselves Poor: A Critical Examination of IV Methods in Marketing Applications , 2014, Mark. Sci..

[69]  D. McCaffrey,et al.  Does alcohol advertising promote adolescent drinking? Results from a longitudinal assessment. , 2005, Addiction.

[70]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[71]  Peter E. Rossi,et al.  The Value of Purchase History Data in Target Marketing , 1996 .

[72]  E. C. Hammond,et al.  Smoking and lung cancer: recent evidence and a discussion of some questions. , 1959, Journal of the National Cancer Institute.

[73]  P. K. Kannan,et al.  From Social to Sale: The Effects of Firm-Generated Content in Social Media on Customer Behavior , 2016 .