Optimizing cluster-based randomized experiments under a monotonicity assumption

Cluster-based randomized experiments are popular designs for mitigating the bias of standard estimators when interference is present and classical causal inference and experimental design assumptions (such as SUTVA or ITR) do not hold. Without an exact knowledge of the interference structure, it can be challenging to understand which partitioning of the experimental units is optimal to minimize the estimation bias. In the paper, we introduce a monotonicity condition under which a novel two-stage experimental design allows us to determine which of two cluster-based designs yields the least biased estimator. We then consider the setting of online advertising auctions and show that reserve price experiments verify the monotonicity condition and the proposed framework and methodology applies. We validate our findings on an advertising auction dataset.

[1]  Jon M. Kleinberg,et al.  Graph cluster randomization: network exposure to multiple universes , 2013, KDD.

[2]  Anmol Bhasin,et al.  Network A/B Testing: From Sampling to Estimation , 2015, WWW.

[3]  H. Varian,et al.  The VCG Auction in Theory and Practice , 2014 .

[4]  Nico Brooks The Atlas rank report, part 2: How search engine rank impacts conversion , 2004 .

[5]  Joel Nishimura,et al.  Restreaming graph partitioning: simple versatile algorithms for advanced balancing , 2013, KDD.

[6]  Diane Lambert,et al.  Randomization and The Pernicious Effects of Limited Budgets on Auction Experiments , 2016, AISTATS.

[7]  Dean Eckles,et al.  Estimating peer effects in networks with peer encouragement designs , 2016, Proceedings of the National Academy of Sciences.

[8]  Dean Eckles,et al.  Design and Analysis of Experiments in Networks: Reducing Bias from Interference , 2014, ArXiv.

[9]  Charalampos E. Tsourakakis,et al.  FENNEL: streaming graph partitioning for massive scale graphs , 2014, WSDM.

[10]  Jean Pouget-Abadie,et al.  Testing for arbitrary interference on experimentation platforms , 2017, Biometrika.

[11]  Vahab S. Mirrokni,et al.  GeoCUTS: Geographic Clustering Using Travel Statistics , 2016, ArXiv.

[12]  Joel A. Middleton,et al.  Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments , 2015 .

[13]  Dylan Walker,et al.  Design of Randomized Experiments in Networks , 2014, Proceedings of the IEEE.

[14]  Konstantin Andreev,et al.  Balanced Graph Partitioning , 2004, SPAA '04.

[15]  Matthew Richardson,et al.  Predicting clicks: estimating the click-through rate for new ads , 2007, WWW '07.

[16]  Edoardo M. Airoldi,et al.  Detecting Network Effects: Randomizing Over Randomized Experiments , 2017, KDD.

[17]  Gabriel Kliot,et al.  Streaming graph partitioning for large distributed graphs , 2012, KDD.

[18]  A. Donner,et al.  Pitfalls of and controversies in cluster randomization trials. , 2004, American journal of public health.

[19]  D. Rubin,et al.  Causal Inference for Statistics, Social, and Biomedical Sciences: Sensitivity Analysis and Bounds , 2015 .

[20]  Lars Backstrom,et al.  Balanced label propagation for partitioning massive graphs , 2013, WSDM.