Report : Estimation and Identification of the Complier Average Causal Effect Parameter in Education RCTs

In randomized control trials (RCTs) in the education field, the complier average causal effect (CACE) parameter is often of policy interest, because it pertains to intervention effects for students who receive a meaningful dose of treatment services. This article uses a causal inference and instrumental variables framework to examine the identification and estimation of the CACE parameter for two-level clustered RCTs. The article also provides simple asymptotic variance formulas for CACE impact estimators measured in nominal and standard deviation units. In the empirical work, data from 10 large RCTs are used to compare significance findings using correct CACE variance estimators and commonly used approximations that ignore the estimation error in service receipt rates and outcome standard deviations. The key finding is that the variance corrections have very little effect on the standard errors of standardized CACE impact estimators. Across the examined outcomes, the correction terms typically raise the standard errors by less than 1%, and change p values at the fourth or higher decimal place. Manuscript received April 16, 2010 Revision received April 26, 2010 Accepted May 24, 2010

[1]  Xihong Lin,et al.  A Comparison of Methods for Estimating the Causal Effect of a Treatment in Randomized Clinical Trials Subject to Noncompliance , 2009, Biometrics.

[2]  L. Hedges Distribution Theory for Glass's Estimator of Effect size and Related Estimators , 1981 .

[3]  Steven Glazerman,et al.  The Effects of Teach For America on Students: Findings from a National Evaluation , 2004 .

[4]  S. Zeger,et al.  Longitudinal data analysis using generalized linear models , 1986 .

[5]  W. Nicholson,et al.  Breaking the Cycle of Poverty: The Effectiveness of Mandatory Services for Welfare-Dependent Teenage Parents , 1993 .

[6]  Allan Donner,et al.  Design and Analysis of Cluster Randomization Trials in Health Research , 2001 .

[7]  Elizabeth A. Stuart,et al.  Closing the Reading Gap: First Year Findings from a Randomized Trial of Four Reading Interventions for Striving Readers , 2006 .

[8]  Peter Z. Schochet Statistical Power for Random Assignment Evaluations of Education Programs , 2005 .

[9]  Howard S. Bloom,et al.  Accounting for No-Shows in Experimental Evaluation Designs , 1984 .

[10]  Jonathan H. Wright,et al.  A Survey of Weak Instruments and Weak Identification in Generalized Method of Moments , 2002 .

[11]  James C. Ohls,et al.  The Effects of Cash-Out on Food Use by Food Stamp Program Participants in San Diego. Princeton, NJ: Mathematica Policy Research , 1992 .

[12]  Tihomir Asparouhov,et al.  Intention‐to‐treat analysis in cluster randomized trials with noncompliance , 2008, Statistics in medicine.

[13]  Peter Z. Schochet,et al.  National Job Corps Study: The Impacts of Job Corps on Participants' Employment and Related Outcomes [and] Methodological Appendixes on the Impact Analysis. , 2001 .

[14]  Peter Z. Schochet,et al.  Does Job Corps Work? Impact Findings from the National Job Corps Study , 2008 .

[15]  David A. Freedman,et al.  On regression adjustments to experimental data , 2008, Adv. Appl. Math..

[16]  Tihomir Asparouhov,et al.  Cluster randomized trials with treatment noncompliance. , 2008, Psychological methods.

[17]  Edward Britton,et al.  Impacts of Comprehensive Teacher Induction: Results from the Second Year of a Randomized Controlled Study , 2009 .

[18]  R. Littell SAS System for Mixed Models , 1996 .

[19]  S. Raudenbush,et al.  Evaluating Kindergarten Retention Policy , 2006 .

[20]  Joshua D. Angrist,et al.  Identification of Causal Effects Using Instrumental Variables , 1993 .

[21]  Jacob Cohen Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[22]  Impacts of Comprehensive Teacher Induction: Results from the First Year of a Randomized Controlled Study. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education , 2008 .

[23]  P. Holland Statistics and Causal Inference , 1985 .

[24]  D. Rubin Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[25]  Cheri A. Vogel,et al.  Making a Difference in the Lives of Infants and Toddlers and Their Families: The Impacts of Early Head Start. Volume II: Final Technical Report Appendixes , 2002 .

[26]  David M. Murray,et al.  Design and Analysis of Group- Randomized Trials , 1998 .

[27]  Mark Dynarski,et al.  When Schools Stay Open Late: The National Evaluation of the 21st-Century Community Learning Centers Program, First Year Findings , 2005 .

[28]  Kosuke Imai,et al.  Survey Sampling , 1998, Nov/Dec 2017.

[29]  Jeffrey A. Smith,et al.  Accounting for Dropouts in Evaluations of Social Experiments , 1994 .

[30]  Michael P. Murray Avoiding Invalid Instruments and Coping with Weak Instruments , 2006 .

[31]  A. Winsor Sampling techniques. , 2000, Nursing times.

[32]  Peter Z. Schochet,et al.  Technical Methods Report: Estimation and Identification of the Complier Average Causal Effect Parameter in Education RCTs. NCEE 2009-4040. , 2009 .

[33]  William R. Penuel,et al.  Effectiveness of Reading and Mathematics Software Products: Findings from the First Student Cohort , 2007 .

[34]  D. Zucker Design and Analysis of Cluster Randomization Trials , 2003 .

[35]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[36]  L. Hedges Effect Sizes in Cluster-Randomized Designs , 2007 .

[37]  Peter Z. Schochet Is regression adjustment supported by the Neyman model for causal inference , 2007 .

[38]  William G. Howell,et al.  School Choice in New York City After Three Years: An Evaluation of the School Choice Scholarships Program , 2002 .

[39]  D. Rubin Assignment to Treatment Group on the Basis of a Covariate , 1976 .

[40]  Paul E. Peterson,et al.  School Choice in New York City After Two Years: An Evaluation of the School Choice Scholarships Program. Interim Report. Washington, DC: Mathematica Policy Research , 2000 .