Cluster-Adaptive Network A/B Testing: From Randomization to Estimation

A/B testing is an important decision-making tool in product development for evaluating user engagement or satisfaction from a new service, feature or product. The goal of A/B testing is to estimate the average treatment effects (ATE) of a new change, which becomes complicated when users are interacting. When the important assumption of A/B testing, the Stable Unit Treatment Value Assumption (SUTVA), which states that each individual's response is affected by their own treatment only, is not valid, the classical estimate of the ATE usually leads to a wrong conclusion. In this paper, we propose a cluster-adaptive network A/B testing procedure, which involves a sequential cluster-adaptive randomization and a cluster-adjusted estimator. The cluster-adaptive randomization is employed to minimize the cluster-level Mahalanobis distance within the two treatment groups, so that the variance of the estimate of the ATE can be reduced. In addition, the cluster-adjusted estimator is used to eliminate the bias caused by network interference, resulting in a consistent estimation for the ATE. Numerical studies suggest our cluster-adaptive network A/B testing achieves consistent estimation with higher efficiency. An empirical study is conducted based on a real world network to illustrate how our method can benefit decision-making in application.

[1]  N. L. Johnson,et al.  Linear Statistical Inference and Its Applications , 1966 .

[2]  Anmol Bhasin,et al.  Network A/B Testing: From Sampling to Estimation , 2015, WWW.

[3]  Calyampudi R. Rao,et al.  Linear Statistical Inference and Its Applications. , 1975 .

[4]  D. Rubin,et al.  Causal Inference for Statistics, Social, and Biomedical Sciences: A General Method for Estimating Sampling Variances for Standard Estimators for Average Causal Effects , 2015 .

[5]  Jun Shao,et al.  A theory for testing hypotheses under covariate-adaptive randomization , 2010 .

[6]  David A. Bader,et al.  Graph Partitioning and Graph Clustering , 2013 .

[7]  Ron Kohavi,et al.  Online controlled experiments at large scale , 2013, KDD.

[8]  Feifang Hu,et al.  Testing Hypotheses of Covariate-Adaptive Randomized Clinical Trials , 2015 .

[9]  Ron Kohavi,et al.  Seven rules of thumb for web site experimenters , 2014, KDD.

[10]  Azeem M. Shaikh,et al.  Inference Under Covariate-Adaptive Randomization , 2015, Journal of the American Statistical Association.

[11]  Dean Eckles,et al.  Design and Analysis of Experiments in Networks: Reducing Bias from Interference , 2014, ArXiv.

[12]  Kari Lock Morgan,et al.  Rerandomization to improve covariate balance in experiments , 2012, 1207.5625.

[13]  S. Raudenbush.  Statistical analysis and optimal design for cluster randomized trials , 1997 .

[14]  Yang Li,et al.  Pairwise Sequential Randomization and Its Properties , 2016, 1611.02802.

[15]  Jon M. Kleinberg,et al.  Graph cluster randomization: network exposure to multiple universes , 2013, KDD.

[16]  P. Aronow,et al.  Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments , 2011 .

[17]  F. Hu,et al.  Asymptotic properties of covariate-adaptive randomization , 2012, 1210.4666.

[18]  Bai Jiang,et al.  A Framework for Network AB Testing , 2016 .

[19]  M E J Newman,et al.  Modularity and community structure in networks. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[20]  D R Taves,et al.  Minimization: A new method of assigning patients to treatment and control groups , 1974, Clinical pharmacology and therapeutics.

[21]  Ryan A. Rossi,et al.  The Network Data Repository with Interactive Graph Analytics and Visualization , 2015, AAAI.

[22]  Alex Pentland,et al.  Reality mining: sensing complex social systems , 2006, Personal and Ubiquitous Computing.

[23]  S. Pocock,et al.  Sequential treatment assignment with balancing for prognostic factors in the controlled clinical trial. , 1975, Biometrics.

[24]  Maneesh Varshney,et al.  Using Ego-Clusters to Measure Network Effects at LinkedIn , 2019, ArXiv.

[25]  Réka Albert,et al.  Near linear time algorithm to detect community structures in large-scale networks. , 2007, Physical review. E, Statistical, nonlinear, and soft matter physics.

[26]  Richard L. Tweedie,et al.  Markov Chains and Stochastic Stability , 1993, Communications and Control Engineering Series.