IDENTIFYING MECHANISMS BEHIND POLICY INTERVENTIONS VIA CAUSAL MEDIATION ANALYSIS

Causal analysis in program evaluation has primarily focused on the question about whether or not a program, or package of policies, has an impact on the targeted outcome of interest. However, it is often of scientific and practical importance to also explain why such impacts occur. In this paper, we introduce causal mediation analysis, a statistical framework for analyzing causal mechanisms that has become increasingly popular in social and medical sciences in recent years. The framework enables us to show exactly what assumptions are sufficient for identifying causal mediation effects for the mechanisms of interest, derive a general algorithm for estimating such mechanism‐specific effects, and formulate a sensitivity analysis for the violation of those identification assumptions. We also discuss an extension of the framework to analyze causal mechanisms in the presence of treatment noncompliance, a common problem in randomized evaluation studies. The methods are illustrated via applications to two intervention studies on pre‐school classes and job‐training workshops.

[1]  Mi Ypsilanti,et al.  HighScope Educational Research Foundation. , 2003 .

[2]  P. Holland.  CAUSAL INFERENCE, PATH ANALYSIS AND RECURSIVE STRUCTURAL EQUATIONS MODELS , 1988 .

[3]  Jane Waldfogel,et al.  Differential effects of high‐quality child care , 2002 .

[4]  J. Robins,et al.  Alternative Graphical Causal Models and the Identification of Direct E!ects , 2010 .

[5]  John W. Payne,et al.  How important is information format? An experimental study of home energy audit programs , 1986 .

[6]  D. Culler,et al.  Comparison of methods , 2000 .

[7]  M. Sobel.  Identification of Causal Parameters in Randomized Studies With Mediating Variables , 2008 .

[8]  J. Heckman,et al.  NBER WORKING PAPER SERIES UNDERSTANDING THE MECHANISMS THROUGH WHICH AN INFLUENTIAL EARLY CHILDHOOD PROGRAM BOOSTED ADULT OUTCOMES , 2012 .

[9]  Michael Puma,et al.  The national evaluation of the food stamp employment and training program , 1994 .

[10]  Angus Deaton.  Instruments, Randomization, and Learning about Development , 2010 .

[11]  G. Galster.  The Mechanism(s) of Neighbourhood Effects: Theory, Evidence, and Policy Implications , 2012 .

[12]  J. Pearl.  Interpretation and Identification of Causal Mediation , 2013, Psychological methods.

[13]  S. West,et al.  A comparison of methods to test mediation and other intervening variable effects. , 2002, Psychological methods.

[14]  Jan M. Rabaey,et al.  Comparison of Methods , 2004 .

[15]  Tyler J VanderWeele,et al.  A three-way decomposition of a total effect into direct, indirect, and interactive effects. , 2013, Epidemiology.

[16]  L. Keele,et al.  A General Approach to Causal Mediation Analysis , 2010, Psychological methods.

[17]  L. Keele,et al.  Identification, Inference and Sensitivity Analysis for Causal Mediation Effects , 2010, 1011.1079.

[18]  J. Robins,et al.  Invited commentary: ecologic studies--biases, misconceptions, and counterexamples. , 1994, American journal of epidemiology.

[19]  R. Kneller.  Quantitative economics 1 , 2017 .

[20]  Paul R Rosenbaum,et al.  Attributing Effects to Treatment in Matched Observational Studies , 2002 .

[21]  Kosuke Imai,et al.  mediation: R Package for Causal Mediation Analysis , 2014 .

[22]  L. Schweinhart,et al.  Effects of the Perry Preschool Program on Youths Through Age 15 , 1981 .

[23]  Joshua D. Angrist,et al.  Identification of Causal Effects Using Instrumental Variables , 1993 .

[24]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[25]  A. Vinokur,et al.  Mastery and inoculation against setbacks as active ingredients in the JOBS intervention for the unemployed. , 1997, Journal of consulting and clinical psychology.

[26]  Alfonso Flores-Lagunes,et al.  Identification and Estimation of Causal Mechanisms and Net Effects of a Treatment Under Unconfoundedness , 2009, SSRN Electronic Journal.

[27]  Lindsay C. Page.  Principal Stratification as a Framework for Investigating Mediational Processes in Experimental Settings , 2012 .

[28]  T. Speed,et al.  On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9 , 1990 .

[29]  M. Zanna,et al.  Establishing a causal chain: why experiments are often more effective than mediational analyses in examining psychological processes. , 2005, Journal of personality and social psychology.

[30]  Elizabeth R. Word.  The State of Tennessee's Student/Teacher Achievement Ratio (STAR) Project: Technical Report (1985-1990). , 1990 .

[31]  David Collier,et al.  Rethinking Social Inquiry: Diverse Tools, Shared Standards , 2004 .

[32]  Peter Green,et al.  Highly Structured Stochastic Systems , 2003 .

[33]  Tyler J. VanderWeele,et al.  Comments: Should Principal Stratification Be Used to Study Mediational Processes? , 2012, Journal of research on educational effectiveness.

[34]  P. Holland.  Statistics and Causal Inference , 1985 .

[35]  Kosuke Imai,et al.  Identification and Sensitivity Analysis for Multiple Causal Mechanisms: Revisiting Evidence from Framing Experiments , 2013, Political Analysis.

[36]  Jeffrey M Albert,et al.  Mediation analysis via potential outcomes models , 2008, Statistics in medicine.

[37]  M. Sobel,et al.  Identification Problems in the Social Sciences. , 1996 .

[38]  M. Huber.  IDENTIFYING CAUSAL MECHANISMS (PRIMARILY) BASED ON INVERSE PROBABILITY WEIGHTING , 2014 .

[39]  Charles F. Manski,et al.  Identification for Prediction and Decision , 2008 .

[40]  Teppei Yamamoto,et al.  Identification and Estimation of Causal Mediation Effects with Treatment Noncompliance , 2014 .

[41]  J. Heckman,et al.  The Rate of Return to the High/Scope Perry Preschool Program , 2009, Journal of public economics.

[42]  Howard S. Bloom,et al.  The Core Analytics of Randomized Experiments for Social Research. MDRC Working Papers on Research Methodology. , 2006 .

[43]  Christopher R. Taber,et al.  Selection on Observed and Unobserved Variables: Assessing the Effectiveness of Catholic Schools , 2000, Journal of Political Economy.

[44]  Kosuke Imai,et al.  Comment on Pearl: Practical implications of theoretical results for causal mediation analysis. , 2014, Psychological methods.

[45]  Kurt Weyland,et al.  Rethinking Social Inquiry: Diverse Tools, Shared Standards , 2005, Perspectives on Politics.

[46]  Carolyn S. Breda,et al.  Evidence from project star about class size and student achievement , 1989 .

[47]  D. A. Kenny,et al.  The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. , 1986, Journal of personality and social psychology.

[48]  Kosuke Imai,et al.  Experimental Designs for Identifying Causal Mechanisms (with discussions) , 2013 .

[49]  Jeffrey M Albert,et al.  Generalized Causal Mediation Analysis , 2011, Biometrics.

[50]  Alfonso Flores-Lagunes,et al.  Nonparametric Partial Identification of Causal Net and Mechanism Average Treatment Effects , 2010 .

[51]  Booil Jo,et al.  Causal inference in randomized experiments with mediational processes. , 2008, Psychological methods.

[52]  Martin Huber,et al.  Identifying causal mechanisms in experiments (primarily) based on inverse probability weighting , 2012 .

[53]  Brian Kisida,et al.  School Vouchers and Student Outcomes: Experimental Evidence from Washington, DC , 2013 .

[54]  Chen Avin,et al.  Identifiability of Path-Specific Effects , 2005, IJCAI.

[55]  Jeffrey R. Kling,et al.  Mechanism Experiments and Policy Evaluations , 2011 .

[56]  Michael R. Sperling,et al.  Editorial comments , 2011, Journal of Neural Transmission.

[57]  Hrishikesh D. Vinod,et al.  Advances in social science research using R , 2010 .

[58]  P. Rosenbaum.  Covariance Adjustment in Randomized Experiments and Observational Studies , 2002 .

[59]  M. Sobel,et al.  Identification Problems in the Social Sciences , 1996 .

[60]  D. Mackinnon.  Introduction to Statistical Mediation Analysis , 2008 .

[61]  Marianne Simonsen,et al.  The costs of motherhood: an analysis using matching estimators , 2006 .

[62]  D. Rubin.  Matched Sampling for Causal Effects: Matching to Remove Bias in Observational Studies , 1973 .

[63]  Kosuke Imai,et al.  Experimental designs for identifying causal mechanisms , 2013 .

[64]  A. Vinokur,et al.  Impact of the JOBS intervention on unemployed workers varying in risk for depression , 1995, American journal of community psychology.

[65]  Jennifer D. Shapka,et al.  Effects of a Girls-Only Curriculum During Adolescence: Performance, Persistence, and Engagement in Mathematics and Science , 2003 .

[66]  Christopher Winship,et al.  Unpacking Neighborhood Influences on Education Outcomes: Setting the Stage for Future Research , 2010 .

[67]  P Skrabanek,et al.  The emptiness of the black box. , 1994, Epidemiology.

[68]  J. Robins,et al.  Identifiability and Exchangeability for Direct and Indirect Effects , 1992, Epidemiology.

[69]  Ann S. Epstein,et al.  Effects of the Perry Preschool Program on Youths Through Age 19 , 1984 .

[70]  J. Heckman,et al.  Analyzing Social Experiments as Implemented: A Reexamination of the Evidence from the Highscope Perry Preschool Program , 2010, Quantitative economics.

[71]  Judea Pearl,et al.  Direct and Indirect Effects , 2001, UAI.

[72]  Jason Wittenberg,et al.  Making the Most Of Statistical Analyses: Improving Interpretation and Presentation , 2000 .

[73]  Kosuke Imai,et al.  Unpacking the Black Box of Causality: Learning about Causal Mechanisms from Experimental and Observational Studies , 2011, American Political Science Review.

[74]  James J. Heckman,et al.  Econometric Mediation Analyses: Identifying the Sources of Treatment Effects from Experimentally Estimated Production Technologies with Unmeasured and Mismeasured Inputs , 2013, Econometric reviews.

[75]  J. Finn,et al.  Answers and Questions About Class Size: A Statewide Experiment , 1990 .

[76]  James J. Heckman,et al.  Assessing the Case for Social Experiments , 1995 .

[77]  C. Judd,et al.  When moderation is mediated and mediation is moderated. , 2005, Journal of personality and social psychology.

[78]  D. Mackinnon,et al.  Equivalence of the Mediation, Confounding and Suppression Effect , 2000, Prevention Science.

[79]  D. Weikart,et al.  The Ypsilanti Perry Preschool Project: Preschool Years and Longitudinal Results Through Fourth Grade. Monograph Number 3. , 1978 .

[80]  Angus Deaton,et al.  Understanding the Mechanisms of Economic Development , 2010 .

[81]  Alan B. Krueger.  Experimental Estimates of Education Production Functions , 1999 .

[82]  Klaus Reinhardt,et al.  Advances In Social Science Research Using R , 2016 .

[83]  G. W. Imbens.  Sensitivity to Exogeneity Assumptions in Program Evaluation , 2003 .

[84]  Kosuke Imai,et al.  Causal Mediation Analysis Using R , 2010 .

[85]  D. Rubin.  [On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.] Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies , 1990 .