Learning About and From a Distribution of Program Impacts Using Multisite Trials

The present article provides a synthesis of the conceptual and statistical issues involved in using multisite randomized trials to learn about and from a distribution of heterogeneous program impacts across individuals and/or program sites. Learning about such a distribution involves estimating its mean value, detecting and quantifying its variation, and estimating site-specific impacts. Learning from such a distribution involves studying the factors that predict or explain impact variation. Part I of the article introduces the concepts and issues involved. Part II focuses on estimating the mean and variation of impacts of program assignment. Part III extends the discussion to variation in the impacts of program participation. Part IV considers how to use multisite trials to study moderators of program impacts (individual-level or site-level factors that influence these impacts) and mediators of program impacts (individual-level or site-level “mechanisms” that produce these impacts).

[1]  R. Fisher.  Statistical methods for research workers , 1927, Protoplasma.

[2]  C. Morris.  Parametric Empirical Bayes Inference: Theory and Applications , 1983 .

[3]  D. Lindley,et al.  Bayes Estimates for the Linear Model , 1972 .

[4]  D. Rubin,et al.  Estimation in Covariance Components Models , 1981 .

[5]  Russell W. Rumberger,et al.  Dropping Out of Middle School: A Multilevel Analysis of Students and Schools , 1995 .

[6]  A. Roy.  Some thoughts on the distribution of earnings , 1951 .

[7]  Donald B. Rubin,et al.  Bayesian Inference for Causal Effects: The Role of Randomization , 1978 .

[8]  Judea Pearl,et al.  Direct and Indirect Effects , 2001, UAI.

[9]  T. Speed,et al.  On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9 , 1990 .

[10]  Anthony S. Bryk,et al.  Heterogeneity of Variance in Experimental Studies: A Challenge to Conventional Interpretations , 1988 .

[11]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[12]  D. Rubin.  Statistics and Causal Inference: Comment: Which Ifs Have Causal Answers , 1986 .

[13]  J. Heckman.  Sample selection bias as a specification error , 1979 .

[14]  L. Keele,et al.  Identification, Inference and Sensitivity Analysis for Causal Mediation Effects , 2010, 1011.1079.

[15]  J. Böhnke.  Explanation in causal inference: Methods for mediation and interaction. , 2016, Quarterly journal of experimental psychology.

[16]  D. Rubin.  Comment: Which Ifs Have Causal Answers , 1986 .

[17]  Sean F. Reardon,et al.  Bias and Bias Correction in Multisite Instrumental Variables Analysis of Heterogeneous Mediator Effects , 2013 .

[18]  P. Holland.  CAUSAL INFERENCE, PATH ANALYSIS AND RECURSIVE STRUCTURAL EQUATIONS MODELS , 1988 .

[19]  H. Bloom,et al.  Quantifying Variation in Head Start Effects on Young Children's Cognitive and Socio-Emotional Skills Using Data from the National Head Start Impact Study , 2015 .

[20]  James J. Heckman,et al.  Instrumental Variables Methods for the Correlated Random Coefficient Model: Estimating the Average Rate of Return to Schooling When the Return is Correlated with Schooling , 1998 .

[21]  Howard S. Bloom,et al.  Linking program implementation and effectiveness: Lessons from a pooled sample of welfare‐to‐work experiments , 2003 .

[22]  David H. Greenberg,et al.  Digest of Social Experiments , 2004 .

[23]  D. Rubin,et al.  Principal Stratification in Causal Inference , 2002, Biometrics.

[24]  C. Glymour,et al.  STATISTICS AND CAUSAL INFERENCE , 1985 .

[25]  J. Robins,et al.  Identifiability and Exchangeability for Direct and Indirect Effects , 1992, Epidemiology.

[26]  Sean F. Reardon,et al.  Under What Assumptions Do Site-by-Treatment Instruments Identify Average Causal Effects? , 2013 .

[27]  T. Louis.  Estimating a population of parameter values using Bayes and empirical Bayes methods , 1984 .

[28]  Sean F. Reardon,et al.  Statistical Analysis for Multisite Trials Using Instrumental Variables With Random Coefficients , 2012 .

[29]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[30]  Michael J. Weiss,et al.  A CONCEPTUAL FRAMEWORK FOR STUDYING THE SOURCES OF VARIATION IN PROGRAM EFFECTS: Sources of Variation in Program Effects , 2014 .

[31]  Jessaca K. Spybrook.  Detecting Intervention Effects Across Context: An Examination of the Precision of Cluster Randomized Trials , 2014 .

[32]  S. Raudenbush,et al.  Evaluating Kindergarten Retention Policy , 2006 .

[33]  D. Katz.  The American Statistical Association , 2000 .