Causal Meta-Mediation Analysis: Inferring Dose-Response Function From Summary Statistics of Many Randomized Experiments

It is common in the internet industry to use offline-developed algorithms to power online products that contribute to the success of a business. Offline-developed algorithms are guided by offline evaluation metrics, which are often different from online business key performance indicators (KPIs). To maximize business KPIs, it is important to pick a north star among all available offline evaluation metrics. By noting that online products can be measured by online evaluation metrics, the online counterparts of offline evaluation metrics, we decompose the problem into two parts. As the offline A/B test literature works out the first part: counterfactual estimators of offline evaluation metrics that move the same way as their online counterparts, we focus on the second part: causal effects of online evaluation metrics on business KPIs. The north star of offline evaluation metrics should be the one whose online counterpart causes the most significant lift in the business KPI. We model the online evaluation metric as a mediator and formalize its causality with the business KPI as dose-response function (DRF). Our novel approach, causal meta-mediation analysis, leverages summary statistics of many existing randomized experiments to identify, estimate, and test the mediator DRF. It is easy to implement and to scale up, and has many advantages over the literature of mediation analysis and meta-analysis. We demonstrate its effectiveness by simulation and implementation on real data.

[1]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[2]  Tommy Stanley,et al.  Meta-Regression Analysis in Economics and Business , 2012 .

[3]  L. Keele,et al.  Identification, Inference and Sensitivity Analysis for Causal Mediation Effects , 2010, 1011.1079.

[4]  Kosuke Imai,et al.  Comment on Pearl: Practical implications of theoretical results for causal mediation analysis. , 2014, Psychological methods.

[5]  G. Imbens.  The Role of the Propensity Score in Estimating Dose-Response Functions , 1999 .

[6]  J. Robins,et al.  Identifiability and Exchangeability for Direct and Indirect Effects , 1992, Epidemiology.

[7]  M. Sobel.  Identification of Causal Parameters in Randomized Studies With Mediating Variables , 2008 .

[8]  D. A. Kenny,et al.  The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. , 1986, Journal of personality and social psychology.

[9]  Kristopher J Preacher,et al.  Mediation Analysis in Social Psychology: Current Practices and New Recommendations , 2011 .

[10]  Dylan S. Small.  Mediation Analysis Without Sequential Ignorability: Using Baseline Covariates Interacted with Random Assignment as Instrumental Variables , 2011, 1109.1070.

[11]  Liangjie Hong,et al.  The Identification and Estimation of Direct and Indirect Effects in A/B Tests through Causal Mediation Analysis , 2019, KDD.

[12]  Jeffrey M. Woodbridge.  Econometric Analysis of Cross Section and Panel Data , 2002 .

[13]  J. Pearl.  Interpretation and Identification of Causal Mediation , 2013, Psychological methods.

[14]  G. Imbens,et al.  The Propensity Score with Continuous Treatments , 2005 .

[15]  Donald B. Rubin,et al.  Basic Concepts of Statistical Inference for Causal Effects in Experiments and Observational Studies , 2004 .

[16]  L. Hedges,et al.  The Handbook of Research Synthesis and Meta-Analysis , 2009 .

[17]  Thomas Nedelec,et al.  Offline A/B Testing for Recommender Systems , 2018, WSDM.

[18]  Donald P. Green,et al.  Enough Already about “Black Box” Experiments: Studying Mediation Is More Difficult than Most Scholars Suppose , 2009 .

[19]  Matthew S. Fritz,et al.  Mediation analysis. , 2019, Annual review of psychology.

[20]  J. Angrist,et al.  Journal of Economic Perspectives—Volume 15, Number 4—Fall 2001—Pages 69–85 Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments , 2022 .

[21]  C. Blanco,et al.  Causality and Psychopathology: Finding the Determinants of Disorders and Their Cures , 2011 .

[22]  James J. Heckman,et al.  Econometric Mediation Analyses: Identifying the Sources of Treatment Effects from Experimentally Estimated Production Technologies with Unmeasured and Mismeasured Inputs , 2013, Econometric reviews.

[23]  Judea Pearl,et al.  Direct and Indirect Effects , 2001, UAI.

[24]  Dean Eckles,et al.  Learning Causal Effects From Many Randomized Experiments Using Regularized Instrumental Variables , 2017, WWW.

[25]  Joshua D. Angrist,et al.  Identification of Causal Effects Using Instrumental Variables , 1993 .

[26]  Judea Pearl,et al.  Reply to commentary by Imai, Keele, Tingley, and Yamamoto concerning causal mediation analysis. , 2014, Psychological methods.