You Can't Always Get What You Want: Using 'Broken Lotteries' to Check the Validity of Charter School Evaluations Using Matching Designs

We consider situations in which public charter school lotteries are neither universally conducted nor consistently documented. Such lotteries produce “broken” Randomized Control Trials, but provide opportunities to assess the internal validity of quasi-experimental research designs. Here, we present the results of a statewide charter school evaluation using a broad-based student matching evaluation design, and run two additional analyses using the charter application waitlists as robustness checks. Our additional models, which address concerns of self-selection by using only charter applicants as matched comparison students, yield similar effect estimates and thus provide support for the use of matching designs in charter school evaluations.

[1]  Jennifer Hill,et al.  A Broader Template for Analyzing Broken Randomized Experiments , 1998 .

[2]  F. Mosteller,et al.  Evidence matters : randomized trials in education research , 2002 .

[3]  Frederic L. Pryor.  Capitalism and freedom , 2010 .

[4]  R. Bifulco,et al.  CAN NONEXPERIMENTAL ESTIMATES REPLICATE ESTIMATES BASED ON RANDOM ASSIGNMENT IN EVALUATIONS OF SCHOOL CHOICE? A WITHIN-STUDY COMPARISON , 2012 .

[5]  P. Gleason,et al.  Using an Experimental Evaluation of Charter Schools to Test Whether Nonexperimental Comparison Group Methods Can Replicate Experimental Impact Estimates , 2012 .

[6]  John F. Kain,et al.  Charter School Quality and Parental Decision Making With School Choice , 2005 .

[7]  E. Tufte.  Beautiful Evidence , 2006 .

[8]  Joshua D. Angrist,et al.  Mostly Harmless Econometrics: An Empiricist's Companion , 2008 .

[9]  Sean F. Reardon.  Review of "How New York City's Charter Schools Affect Achievement". , 2009 .

[10]  Chester E. Finn,et al.  Charter Schools , 1999 .

[11]  Marcus A. Winters,et al.  Apples to Apples: An Evaluation of Charter Schools Serving General Student Populations. Education Working Paper No. 1. , 2003 .

[12]  J. Bohte.  Examining the Impact of Charter Schools on Performance in Traditional Public Schools , 2004 .

[13]  Sarah R. Cohodes,et al.  Informing the Debate: Comparing Boston's Charter, Pilot and Traditional Schools , 2009 .

[14]  H. White.  A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity , 1980 .

[15]  B. Applegate,et al.  Review of Multiple Choice: Charter School Performance in 16 States , 2009 .

[16]  J. Betts,et al.  A Meta-Analysis of the Literature on the Effect of Charter Schools on Student Achievement. , 2016 .

[17]  P. Gleason,et al.  Do Charter Schools Improve Student Achievement? Evidence from a National Randomized Study. Working Paper. , 2011 .

[18]  P. Gleason,et al.  Do Charter Schools Improve Student Achievement? , 2015 .

[19]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[20]  Carl Franklin.  Rhetoric versus Reality: What We Know and What We Need to Know about Vouchers and Charter Schools , 2002 .

[21]  Donald B. Rubin,et al.  Principal Stratification Approach to Broken Randomized Experiments , 2003 .

[22]  Scott M. Gilpatric,et al.  The impact of charter school attendance on student performance , 2007 .