The Seasonality Of Paid Search Effectiveness From A Long Running Field Test

Paid Search, also known as Search Engine Marketing (SEM), has been the largest channel in online advertising by revenue. Yet its causal effectiveness has been difficult to measure. At eBay marketplace we have developed a hybrid Geo+User experiment approach, and conducted a long-running field test over 17-month period to understand the incrementality of paid search on one of the largest e-commerce platforms in the U.S. The experiment results indicate that paid search campaigns drive statistically significant sales lift to the eBay site, and a portion of the direct sales made via paid search is incremental. More importantly, such incrementality demonstrates strong seasonality during the year, and do not necessarily correlate with the paid search spend levels. Our results also show that the new user acquisition lift is higher than the immediate sales lift on the eBay site from paid search campaigns. These findings provide critical insights for paid search and overall online advertising investment strategies.

[1]  D. Rubin.  Matched Sampling for Causal Effects: Matching to Remove Bias in Observational Studies , 1973 .

[2]  David H. Reiley,et al.  Online ads and offline sales: measuring the effect of retail advertising via a controlled experiment on Yahoo! , 2014 .

[3]  Jeffrey H Silber,et al.  Optimal multivariate matching before randomization. , 2004, Biostatistics.

[4]  Rong Ge,et al.  Evaluating online ad campaigns in a pipeline: causal models at scale , 2010, KDD.

[5]  Ashish Agarwal,et al.  Overlapping experiment infrastructure: more, better, faster experimentation , 2010, KDD.

[6]  Jack J. Phillips.  Measuring return on investment , 1994 .

[7]  Kenneth R. Koedinger,et al.  The Rise of the Super Experiment , 2012, EDM.

[8]  Michael S. Bernstein,et al.  Designing and deploying online field experiments , 2014, WWW.

[9]  Yuan Yuan,et al.  Incremental Clicks , 2011, Journal of Advertising Research.

[10]  Steven Tadelis,et al.  Consumer Heterogeneity and Paid Search Effectiveness: A Large Scale Field Experiment , 2014 .

[11]  Gary King,et al.  Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference , 2007, Political Analysis.

[12]  Ron Kohavi,et al.  Responsible editor: R. Bayardo. , 2022 .

[13]  D. Freedman,et al.  Asymptotic Normality and the Bootstrap in Stratified Sampling , 1984 .

[14]  C. F. Wu,et al.  Resampling Inference with Complex Survey Data , 1988 .

[15]  Jon Vaver,et al.  Measuring Ad Effectiveness Using Geo Experiments , 2011 .

[16]  Justin M. Rao.  On the Near Impossibility of Measuring the Returns to Advertising , 2013 .

[17]  O. Pons.  Bootstrap of means under stratified sampling , 2007, 0709.3246.

[18]  James K. Brewer,et al.  Statistical Rules of Thumb , 2003 .

[19]  Jon Vaver,et al.  Periodic Measurement of Advertising Effectiveness Using Multiple-Test-Period Geo Experiments , 2012 .

[20]  R. Sitter.  A resampling procedure for complex survey data , 1992 .