Factorial Designs for Online Experiments

Abstract Online experiments and specifically A/B testing are commonly used to identify whether a proposed change to a web page is in fact an effective one. This study focuses on basic settings in which a binary outcome is obtained from each user who visits the website and the probability of a response may be affected by numerous factors. We use Bayesian probit regression to model the factor effects and combine elements from traditional two-level factorial experiments and multiarmed bandits to construct sequential designs that embed attractive features of estimation and exploitation.

[1]  Yuhong Yang,et al.  RANDOMIZED ALLOCATION WITH NONPARAMETRIC ESTIMATION FOR A MULTI-ARMED BANDIT PROBLEM WITH COVARIATES , 2002 .

[2]  Steven L. Scott,et al.  A modern Bayesian look at the multi-armed bandit , 2010 .

[3]  W. R. Thompson.  ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLES , 1933 .

[4]  R. Agrawal.  Sample mean based index policies by O(log n) regret for the multi-armed bandit problem , 1995, Advances in Applied Probability.

[5]  Jason L. Loeppky,et al.  A Survey of Online Experiment Design with the Stochastic Multi-Armed Bandit , 2015, ArXiv.

[6]  D'Arcy Mays,et al.  D-Optimal Design for Network A/B Testing , 2019, Journal of Statistical Theory and Practice.

[7]  H. Robbins,et al.  Asymptotically efficient adaptive allocation rules , 1985 .

[8]  Neeraj Arora,et al.  Sliced Designs for Multi-Platform Online Experiments , 2020, Technometrics.

[9]  A. Albert,et al.  On the existence of maximum likelihood estimates in logistic regression models , 1984 .

[10]  Hovav A. Dror,et al.  Sequential Experimental Designs for Generalized Linear Models , 2008 .

[11]  Michael S. Bernstein,et al.  Designing and deploying online field experiments , 2014, WWW.

[12]  Ron Kohavi,et al.  Responsible editor: R. Bayardo. , 2022 .

[13]  Kevin D. Glazebrook,et al.  Multi-Armed Bandit Allocation Indices: Gittins/Multi-Armed Bandit Allocation Indices , 2011 .

[14]  Susan Athey,et al.  Estimation Considerations in Contextual Bandits , 2017, ArXiv.

[15]  M. Silvapulle.  On the Existence of Maximum Likelihood Estimators for the Binomial Response Models , 1981 .

[16]  P. Whittle.  Restless bandits: activity allocation in a changing world , 1988, Journal of Applied Probability.

[17]  M. G. Pittau,et al.  A weakly informative default prior distribution for logistic and other regression models , 2008, 0901.4011.

[18]  Peter Auer,et al.  Finite-time Analysis of the Multiarmed Bandit Problem , 2002, Machine Learning.

[19]  S. Chib,et al.  Bayesian analysis of binary and polychotomous response data , 1993 .

[20]  Steven L. Scott,et al.  Multi-armed bandit experiments in the online service economy , 2015 .

[21]  Richard W. Hamming,et al.  Error detecting and error correcting codes , 1950 .

[22]  Benjamin Van Roy,et al.  Learning to Optimize via Posterior Sampling , 2013, Math. Oper. Res..