Design and Analysis of Bipartite Experiments Under a Linear Exposure-response Model

A bipartite experiment consists of one set of units being assigned treatments and another set of units for which we measure outcomes. The two sets of units are connected by a bipartite graph, governing how the treated units can affect the outcome units. The bipartite framework naturally arises in marketplace experiments where, for example, experimenters may seek to investigate the effect of discounting goods on buyer behavior. In this paper, we consider estimation of the average total treatment effect in the bipartite experimental framework under a linear exposure-response model. We introduce the Exposure Reweighted Linear (ERL) estimator, and show that the estimator is unbiased, consistent and asymptotically normal, provided that the bipartite graph is sufficiently sparse. To facilitate inference, we introduce an unbiased and consistent estimator of the variance of the ERL point estimator. In addition, we introduce a cluster-based design, Exposure-Design, that uses heuristics to increase the precision of the ERL estimator by realizing a desirable exposure distribution. Finally, we demonstrate the application of the described methodology to marketplace experiments using a publicly available Amazon user-item review dataset. The full version of the paper is available at: https://arxiv.org/abs/2103.06392.

[1]  T. Słoczyński.  Interpreting OLS Estimands When Treatment Effects Are Heterogeneous: Smaller Groups Get Larger Weights , 2018, Review of Economics and Statistics.

[2]  Fredrik Savje.  Causal inference with misspecified exposure mappings , 2021 .

[3]  Drew Dimmery,et al.  Experimentation for Homogenous Policy Change , 2021, 2101.12318.

[4]  Corwin M Zigler,et al.  Bipartite Causal Inference with Interference. , 2018, Statistical science : a review journal of the Institute of Mathematical Statistics.

[5]  Fredrik Sävje,et al.  AVERAGE TREATMENT EFFECTS IN THE PRESENCE OF UNKNOWN INTERFERENCE. , 2017, Annals of statistics.

[6]  Min Liu,et al.  Trustworthy Online Marketplace Experimentation with Budget-split Design , 2020 .

[7]  V. Mirrokni,et al.  Causal Inference with Bipartite Designs , 2020, 2010.02108.

[8]  David Holtz,et al.  Reducing Interference Bias in Online Marketplace Pricing Experiments , 2020, 2004.12489.

[9]  Brian T. Maurer.  Regression. , 2020, JAAPA : official journal of the American Academy of Physician Assistants.

[10]  Ramesh Johari,et al.  Experimental Design in Two-Sided Platforms: An Analysis of Bias , 2020, EC.

[11]  D. Spielman,et al.  Balancing Covariates in Randomized Experiments with the Gram-Schmidt Walk Design , 2019, 1911.03071.

[12]  Ya Xu,et al.  Top Challenges from the first Practical Online Controlled Experiments Summit , 2019, SKDD.

[13]  Alex Chin,et al.  Regression Adjustments for Estimating the Global Treatment Effect in Experiments with Interference , 2018, Journal of Causal Inference.

[14]  Vahab Mirrokni,et al.  Variance Reduction in Bipartite Experiments through Correlation Clustering , 2019, NeurIPS.

[15]  Alex J. Chin,et al.  Central limit theorems via Stein's method for randomized experiments under interference , 2018, 1804.03105.

[16]  J. Aislinn Bohren,et al.  Optimal Design of Experiments in the Presence of Interference , 2017, Review of Economics and Statistics.

[17]  Edoardo Airoldi,et al.  Limitations of Design-based Causal Inference and A/B Testing under Arbitrary and Network Interference , 2017, Sociological Methodology.

[18]  Elizabeth L. Ogburn,et al.  Causal Inference for Social Network Data , 2017, Journal of the American Statistical Association.

[19]  Andrey Fradkin.  Search, Matching, and the Role of Digital Marketplace Design in Enabling Trade: Evidence from Airbnb , 2017 .

[20]  Peter M. Aronow,et al.  Estimating Average Causal Effects Under Interference Between Units , 2013, 1305.6156.

[21]  M. Elizabeth Halloran,et al.  Dependent Happenings: a Recent Methodological Review , 2016, Current Epidemiology Reports.

[22]  Dean Eckles,et al.  Estimating peer effects in networks with peer encouragement designs , 2016, Proceedings of the National Academy of Sciences.

[23]  Diane Lambert,et al.  Randomization and The Pernicious Effects of Limited Budgets on Auction Experiments , 2016, AISTATS.

[24]  Julian J. McAuley,et al.  Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering , 2016, WWW.

[25]  Vahab S. Mirrokni,et al.  Distributed Balanced Partitioning via Linear Embedding , 2015, WSDM.

[26]  Andrey Fradkin,et al.  Search Frictions and the Design of Online Marketplaces , 2015, AMMA 2015.

[27]  Anton van den Hengel,et al.  Image-Based Recommendations on Styles and Substitutes , 2015, SIGIR.

[28]  Cyrus Samii,et al.  Does Regression Produce Representative Estimates of Causal Effects? , 2015 .

[29]  Thomas Blake,et al.  Why marketplace experimentation is harder than it seems: the role of test-control interference , 2014, EC.

[30]  Dean Eckles,et al.  Design and Analysis of Experiments in Networks: Reducing Bias from Interference , 2014, ArXiv.

[31]  Edward K. Kao,et al.  Estimation of Causal Peer Influence Effects , 2013, ICML.

[32]  Peter M. Aronow,et al.  Conservative variance estimation for sampling designs with zero pairwise inclusion probabilities , 2013 .

[33]  Donald P. Green,et al.  Detecting Spillover Effects: Design and Analysis of Multilevel Experiments , 2012 .

[34]  Tyler J VanderWeele,et al.  On causal inference in the presence of interference , 2012, Statistical methods in medical research.

[35]  Nathan Ross.  Fundamentals of Stein's method , 2011, 1109.1880.

[36]  Neel Sundaresan,et al.  Learning from Seller Experiments in Online Markets , 2011 .

[37]  Markku Rahiala,et al.  Panel Data , 2011, International Encyclopedia of Statistical Science.

[38]  Dylan Walker,et al.  Creating Social Contagion Through Viral Product Design: A Randomized Trial of Peer Influence in Networks , 2010, ICIS.

[39]  A. Wirth.  Correlation Clustering , 2010, Encyclopedia of Machine Learning and Data Mining.

[40]  M. Elsner,et al.  Bounding and Comparing Methods for Correlation Clustering Beyond ILP , 2009, ILP 2009.

[41]  M. Hudgens,et al.  Toward Causal Inference With Interference , 2008, Journal of the American Statistical Association.

[42]  Bernard Fortin,et al.  Identification of Peer Effects through Social Networks , 2007, SSRN Electronic Journal.

[43]  Lorenzo Fattorini,et al.  Applying the Horvitz-Thompson criterion in complex designs: A computer-intensive perspective for estimating inclusion probabilities , 2006 .

[44]  Chaitanya Swamy,et al.  Correlation Clustering: maximizing agreements via semidefinite programming , 2004, SODA '04.

[45]  Venkatesan Guruswami,et al.  Clustering with qualitative information , 2003, 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings..

[46]  David Lucking-Reiley.  Field experiments on the effects of reserve prices in auctions: More magic on the internet , 2000 .

[47]  Edith Cohen,et al.  Approximating matrix multiplication for pattern recognition tasks , 1997, SODA '97.

[48]  R. A. Kempton,et al.  Interference between plots , 1997 .

[49]  J. Angrist,et al.  Estimating the Labor Market Impact of Voluntary Military Service Using Social Security Data on Military Applicants , 1995 .

[50]  C. Manski.  Identification of Endogenous Social Effects: The Reflection Problem , 1993 .

[51]  A. Goldberger.  A course in econometrics , 1991 .

[52]  M E Halloran,et al.  The behaviour of common measures of association used to assess a vaccination programme under complex disease transmission patterns--a computer simulation study of malaria vaccines. , 1990, International journal of epidemiology.

[53]  P. Holland.  Statistics and Causal Inference , 1985 .

[54]  D. Horvitz,et al.  A Generalization of Sampling Without Replacement from a Finite Universe , 1952 .

[55]  S. S. Wilks.  CERTAIN GENERALIZATIONS IN THE ANALYSIS OF VARIANCE , 1932 .