Testing for arbitrary interference on experimentation platforms

Experimentation platforms are essential to large modern technology companies, as they are used to carry out many randomized experiments daily. The classic assumption of no interference among users, under which the outcome for one user does not depend on the treatment assigned to other users, is rarely tenable on such platforms. Here, we introduce an experimental design strategy for testing whether this assumption holds. Our approach is in the spirit of the Durbin–Wu–Hausman test for endogeneity in econometrics, where multiple estimators return the same estimate if and only if the null hypothesis holds. The design that we introduce makes no assumptions on the interference model between units, nor on the network among the units, and has a sharp bound on the variance and an implied analytical bound on the Type I error rate. We discuss how to apply the proposed design strategy to large experimentation platforms, and we illustrate it in the context of an experiment on the LinkedIn platform.

[1]  I NICOLETTI,et al.  The Planning of Experiments , 1936, Rivista di clinica pediatrica.

[2]  M. Chao,et al.  Negative Moments of Positive Random Variables , 1972 .

[3]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[4]  J. Hausman.  Specification tests in econometrics , 1978 .

[5]  Jerome Cornfield,et al.  SYMPOSIUM ON CHD PREVENTION TRIALS: DESIGN ISSUES IN TESTING LIFE STYLE INTERVENTIONRANDOMIZATION BY GROUP: A FORMAL ANALYSIS , 1978 .

[6]  Donald B. Rubin,et al.  Formal modes of statistical inference for causal effects , 1990 .

[7]  Community Intervention Trial for Smoking Cessation (COMMIT): summary of design and intervention. COMMIT Research Group. , 1991, Journal of the National Cancer Institute.

[8]  M E Halloran,et al.  Efficiency of Estimating Vaccine Efficacy for Susceptibility and Infectiousness: Randomization by Individual Versus Household , 1999, Biometrics.

[9]  A. Donner,et al.  Pitfalls of and controversies in cluster randomization trials. , 2004, American journal of public health.

[10]  S. Raudenbush,et al.  Evaluating Kindergarten Retention Policy , 2006 .

[11]  P. Rosenbaum.  Interference Between Units in Randomized Experiments , 2007 .

[12]  M. Hudgens,et al.  Toward Causal Inference With Interference , 2008, Journal of the American Statistical Association.

[13]  P. Aronow.  A General Method for Detecting Interference Between Units in Randomized Experiments , 2010 .

[14]  Ashish Agarwal,et al.  Overlapping experiment infrastructure: more, better, faster experimentation , 2010, KDD.

[16]  Dylan Walker,et al.  Creating Social Contagion Through Viral Product Design: A Randomized Trial of Peer Influence in Networks , 2010, ICIS.

[17]  Jon M. Kleinberg,et al.  Network bucket testing , 2011, WWW.

[18]  Donald P. Green,et al.  Detecting Spillover Effects: Design and Analysis of Multilevel Experiments , 2012 .

[19]  Edo Liberty,et al.  Framework and algorithms for network bucket testing , 2012, WWW.

[20]  Tyler J VanderWeele,et al.  On causal inference in the presence of interference , 2012, Statistical methods in medical research.

[21]  Charles F. Manski,et al.  Identification of Treatment Response with Social Interactions , 2013 .

[22]  Joel A. Middleton,et al.  A Class of Unbiased Estimators of the Average Treatment Effect in Randomized Experiments , 2013 .

[23]  Edward K. Kao,et al.  Estimation of Causal Peer Influence Effects , 2013, ICML.

[24]  Jon M. Kleinberg,et al.  Graph cluster randomization: network exposure to multiple universes , 2013, KDD.

[25]  Joel Nishimura,et al.  Restreaming graph partitioning: simple versatile algorithms for advanced balancing , 2013, KDD.

[26]  Ron Kohavi,et al.  Online controlled experiments at large scale , 2013, KDD.

[27]  Designing experiments to measure spillover effects , 2014 .

[28]  Dean Eckles,et al.  Design and Analysis of Experiments in Networks: Reducing Bias from Interference , 2014, ArXiv.

[29]  David S. Choi,et al.  Estimation of Monotone Treatment Effects in Network Experiments , 2014, ArXiv.

[30]  Pasi Fränti,et al.  Balanced K-Means for Clustering , 2014, S+SSPR.

[31]  Michael S. Bernstein,et al.  Designing and deploying online field experiments , 2014, WWW.

[32]  Edoardo M. Airoldi,et al.  Optimal design of experiments in the presence of network-correlated outcomes , 2015, ArXiv.

[33]  D. Rubin,et al.  Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction , 2016 .

[34]  G. Imbens,et al.  Exact p-Values for Network Interference , 2015, 1506.02084.

[35]  D. Rubin,et al.  Causal Inference for Statistics, Social, and Biomedical Sciences: A General Method for Estimating Sampling Variances for Standard Estimators for Average Causal Effects , 2015 .

[36]  Anmol Bhasin,et al.  Network A/B Testing: From Sampling to Estimation , 2015, WWW.

[37]  Anmol Bhasin,et al.  From Infrastructure to Culture: A/B Testing Challenges in Large Scale Social Networks , 2015, KDD.

[38]  P. Aronow,et al.  Estimating Average Causal Effects Under Interference Between Units , 2015 .

[39]  Dean Eckles,et al.  Estimating peer effects in networks with peer encouragement designs , 2016, Proceedings of the National Academy of Sciences.

[40]  Edoardo M. Airoldi,et al.  Detecting Network Effects: Randomizing Over Randomized Experiments , 2017, KDD.

[41]  D. Sussman,et al.  Elements of estimation theory for causal effects in the presence of network interference , 2017, 1702.03578.

[42]  Holly B. Shakya,et al.  Exploiting social influence to magnify population-level behaviour change in maternal and child health: study protocol for a randomised controlled trial of network targeting algorithms in rural Honduras , 2017, BMJ Open.

[43]  E. Airoldi,et al.  A systematic investigation of classical causal inference strategies under mis-specification due to network interference , 2018, 1810.08259.

[44]  Edoardo M. Airoldi,et al.  Model-assisted design of experiments in the presence of network-correlated outcomes , 2015, Biometrika.