Using Ego-Clusters to Measure Network Effects at LinkedIn

A network effect is said to take place when a new feature not only impacts the people who receive it, but also other users of the platform, like their connections or the people who follow them. This very common phenomenon violates the fundamental assumption underpinning nearly all enterprise experimentation systems, the stable unit treatment value assumption (SUTVA). When this assumption is broken, a typical experimentation platform, which relies on Bernoulli randomization for assignment and two-sample t-test for assessment of significance, will not only fail to account for the network effect, but potentially give highly biased results. This paper outlines a simple and scalable solution to measuring network effects, using ego-network randomization, where a cluster is comprised of an "ego" (a focal individual), and her "alters" (the individuals she is immediately connected to). Our approach aims at maintaining representativity of clusters, avoiding strong modeling assumption, and significantly increasing power compared to traditional cluster-based randomization. In particular, it does not require product-specific experiment design, or high levels of investment from engineering teams, and does not require any changes to experimentation and analysis platforms, as it only requires assigning treatment an individual level. Each user either has the feature or does not, and no complex manipulation of interactions between users is needed. It focuses on measuring the one-out network effect (i.e the effect of my immediate connection's treatment on me), and gives reasonable estimates at a very low setup cost, allowing us to run such experiments dozens of times a year.

[1]  Jean Pouget-Abadie,et al.  Testing for arbitrary interference on experimentation platforms , 2017, Biometrika.

[2]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[3]  Edward K. Kao,et al.  Estimation of Causal Peer Influence Effects , 2013, ICML.

[4]  D. Sussman,et al.  Elements of estimation theory for causal effects in the presence of network interference , 2017, 1702.03578.

[5]  Guillaume Saint-Jacques,et al.  A Method for Measuring Network Effects of One-to-One Communication Features in Online A/B Tests , 2019 .

[6]  Edoardo M. Airoldi,et al.  Identification and Estimation of Treatment and Interference Effects in Observational Studies on Networks , 2016, Journal of the American Statistical Association.

[7]  Michael S. Bernstein,et al.  Designing and deploying online field experiments , 2014, WWW.

[8]  P. Aronow,et al.  Estimating Average Causal Effects Under Interference Between Units , 2015 .

[9]  Tyler J VanderWeele,et al.  On causal inference in the presence of interference , 2012, Statistical methods in medical research.

[10]  Edoardo M. Airoldi,et al.  Detecting Network Effects: Randomizing Over Randomized Experiments , 2017, KDD.

[11]  M. McPherson,et al.  Birds of a Feather: Homophily in Social Networks , 2001 .

[12]  Jon M. Kleinberg,et al.  Graph cluster randomization: network exposure to multiple universes , 2013, KDD.

[13]  Ron Kohavi,et al.  Trustworthy online controlled experiments: five puzzling outcomes explained , 2012, KDD.

[14]  Donald B. Rubin,et al.  Formal modes of statistical inference for causal effects , 1990 .

[15]  Peter M. Aronow,et al.  Estimating Average Causal Effects Under Interference Between Units , 2013, 1305.6156.

[16]  Edoardo M. Airoldi,et al.  Optimal design of experiments in the presence of network-correlated outcomes , 2015, ArXiv.

[17]  Donald B. Rubin,et al.  Bayesian Inference for Causal Effects: The Role of Randomization , 1978 .

[18]  M. Hudgens,et al.  Toward Causal Inference With Interference , 2008, Journal of the American Statistical Association.

[19]  Guillaume Saint-Jacques,et al.  Estimating Network Effects Using Naturally Occurring Peer Notification Queue Counterfactuals , 2019, ArXiv.

[20]  Dean Eckles,et al.  Design and Analysis of Experiments in Networks: Reducing Bias from Interference , 2014, ArXiv.

[21]  Ron Kohavi,et al.  Online controlled experiments at large scale , 2013, KDD.

[22]  Anmol Bhasin,et al.  From Infrastructure to Culture: A/B Testing Challenges in Large Scale Social Networks , 2015, KDD.

[23]  I NICOLETTI,et al.  The Planning of Experiments , 1936, Rivista di clinica pediatrica.

[24]  Joel A. Middleton,et al.  A Class of Unbiased Estimators of the Average Treatment Effect in Randomized Experiments , 2013 .

[25]  Cosma Rohilla Shalizi,et al.  Homophily and Contagion Are Generically Confounded in Observational Social Network Studies , 2010, Sociological methods & research.