Network Experimentation at Scale

We describe our framework, deployed at Facebook, that accounts for interference between experimental units through cluster-randomized experiments. We document this system, including the design and estimation procedures, and detail insights we have gained from the many experiments that have used this system at scale. We introduce a cluster-based regression adjustment that substantially improves precision for estimating global treatment effects as well as testing for interference as part of our estimation procedure. With this regression adjustment, we find that imbalanced clusters can better account for interference than balanced clusters without sacrificing accuracy. In addition, we show how logging exposure to a treatment can be used for additional variance reduction. Interference is a widely acknowledged issue with online field experiments, yet there is less evidence from real-world experiments demonstrating interference in online settings. We fill this gap by describing two case studies that capture significant network effects and highlight the value of this experimentation framework.

[1]  Alex Chin,et al.  Regression Adjustments for Estimating the Global Treatment Effect in Experiments with Interference , 2018, Journal of Causal Inference.

[2]  P. Aronow,et al.  Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments , 2011 .

[3]  Huizhi Xie,et al.  Improving the Sensitivity of Online Controlled Experiments: Case Studies at Netflix , 2016, KDD.

[4]  G. Imbens,et al.  Exact p-Values for Network Interference , 2015, 1506.02084.

[5]  D. Rubin.  Randomization Analysis of Experimental Data: The Fisher Randomization Test Comment , 1980 .

[6]  Vahab S. Mirrokni,et al.  Randomized Experimental Design via Geographic Clustering , 2016, KDD.

[7]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[8]  Dean Eckles,et al.  Design and Analysis of Experiments in Networks: Reducing Bias from Interference , 2014, ArXiv.

[9]  M. Hudgens,et al.  Toward Causal Inference With Interference , 2008, Journal of the American Statistical Association.

[10]  Brian Karrer,et al.  Social Hash Partitioner: A Scalable Distributed Hypergraph Partitioner , 2017, Proc. VLDB Endow..

[11]  Anmol Bhasin,et al.  From Infrastructure to Culture: A/B Testing Challenges in Large Scale Social Networks , 2015, KDD.

[12]  J. Pearl.  Causality: Models, Reasoning and Inference , 2000 .

[13]  M E J Newman,et al.  Finding and evaluating community structure in networks. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[14]  Anmol Bhasin,et al.  Network A/B Testing: From Sampling to Estimation , 2015, WWW.

[15]  Joel A. Middleton,et al.  Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments , 2015 .

[16]  Jon M. Kleinberg,et al.  Graph cluster randomization: network exposure to multiple universes , 2013, KDD.

[17]  Peter M. Aronow,et al.  Estimating Average Causal Effects Under Interference Between Units , 2013, 1305.6156.

[18]  Ilias Gerostathopoulos,et al.  Engineering for a Science-Centric Experimentation Platform , 2019, 2020 IEEE/ACM 42nd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[19]  Louis H. Y. Chen,et al.  Normal approximation under local dependence , 2004, math/0410104.

[20]  Ron Kohavi,et al.  Improving the sensitivity of online controlled experiments by utilizing pre-experiment data , 2013, WSDM.

[21]  Richard J. Hayes,et al.  Cluster randomised trials , 2009 .

[22]  W. Lin,et al.  Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique , 2012, 1208.2301.

[23]  David Holtz,et al.  Reducing Interference Bias in Online Marketplace Pricing Experiments , 2020, 2004.12489.

[24]  Jean-Loup Guillaume,et al.  Fast unfolding of communities in large networks , 2008, 0803.0476.

[25]  Edoardo M. Airoldi,et al.  Detecting Network Effects: Randomizing Over Randomized Experiments , 2017, KDD.

[26]  I NICOLETTI,et al.  The Planning of Experiments , 1936, Rivista di clinica pediatrica.

[27]  David R. Cox.  Planning of Experiments , 1958 .

[28]  Alex Deng,et al.  Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas , 2018, KDD.

[29]  Jean Pouget-Abadie,et al.  Testing for arbitrary interference on experimentation platforms , 2017, Biometrika.