Replicating Experimental Impact Estimates with Nonexperimental Methods in the Context of Control Crossover

Ideally, nonexperimental methods that aim to replicate the results of rigorous randomized experiments focus on the intent to treat (ITT) experimental impact estimate, the most causally rigorous measure. This working paper develops a new replication approach that facilitates testing nonexperimental methods against rigorous experimental ITT impact estimates when the experiment includes substantial control crossover.

[1]  P. Gleason,et al.  The Evaluation of Charter School Impacts: Final Report. NCEE 2010-4029. , 2010 .

[2]  R. Murnane,et al.  Methods Matter: Improving Causal Inference in Educational and Social Science Research , 2010 .

[3]  Roland G. Fryer,et al.  Are High-Quality Schools Enough to Increase Achievement Among the Poor? Evidence from the Harlem Children's Zone † , 2011 .

[4]  G. Bohrnstedt,et al.  Class-Size Reduction in California , 2001 .

[5]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[6]  Roland G. Fryer,et al.  Getting Beneath the Veil of Effective Schools: Evidence from New York City , 2011 .

[7]  John Gibson,et al.  How Important is Selection? Experimental Versus Non-Experimental Measures of the Income Gains from Migration , 2006 .

[8]  P. Gleason,et al.  Using an Experimental Evaluation of Charter Schools to Test Whether Nonexperimental Comparison Group Methods Can Replicate Experimental Impact Estimates , 2012 .

[9]  Christopher R. Walters,et al.  Charter Schools and the Road to College Readiness: The Effects on College Preparation, Attendance and Choice. Understanding Boston. , 2013 .

[10]  Parag A. Pathak,et al.  Explaining Charter School Effectiveness , 2011, SSRN Electronic Journal.

[11]  D. Katz.  The American Statistical Association , 2000 .

[12]  D. Brewer,et al.  Rhetoric versus Reality: What We Know and What We Need to Know About Vouchers and Charter Schools , 2007 .

[13]  P. Gleason,et al.  Horseshoes, Hand Grenades, and Treatment Effects? Reassessing Bias in Nonexperimental Estimators , 2013 .

[14]  Jeffrey A. Smith,et al.  Does Matching Overcome Lalonde's Critique of Nonexperimental Estimators? , 2000 .

[15]  R. Bifulco,et al.  CAN NONEXPERIMENTAL ESTIMATES REPLICATE ESTIMATES BASED ON RANDOM ASSIGNMENT IN EVALUATIONS OF SCHOOL CHOICE? A WITHIN-STUDY COMPARISON , 2012 .

[16]  Steven Glazerman,et al.  Nonexperimental Versus Experimental Estimates of Earnings Impacts , 2003 .

[17]  P. Gleason,et al.  Using Lotteries to Evaluate Schools of Choice: Evidence from a National Study of Charter Schools , 2012 .

[18]  Parag A. Pathak,et al.  Accountability and Flexibility in Public Schools: Evidence from Boston's Charters and Pilots , 2009 .

[19]  Vivian C. Wong,et al.  Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons , 2008 .

[20]  C. Hoxby,et al.  Charter Schools in New York City: Who Enrolls and How They Affect Their Students' Achievement , 2009 .

[21]  Natalya Verbitsky-Savitz,et al.  Charter-School Management Organizations: Diverse Strategies and Diverse Student Impacts. Updated Edition. , 2011 .

[22]  P. Gleason,et al.  Student Characteristics and Achievement in 22 KIPP Middle Schools: Final Report. , 2010 .