Scaling Up What Works: Experimental Evidence on External Validity in Kenyan Education

The recent wave of randomized trials in development economics has provoked criticisms regarding external validity. We investigate two concerns – heterogeneity across beneficiaries and implementers – in a randomized trial of contract teachers in Kenyan schools. The intervention, previously shown to raise test scores in NGO-led trials in Western Kenya and parts of India, was replicated across all Kenyan provinces by an NGO and the government. Strong effects of short-term contracts produced in controlled experimental settings are lost in weak public institutions: NGO implementation produces a positive effect on test scores across diverse contexts, while government implementation yields zero effect. The data suggests that the stark contrast in success between the government and NGO arm can be traced back to implementation constraints and political economy forces put in motion as the program went to scale.

[1]  Esther Duflo,et al.  Peer Effects, Teacher Incentives, and the Impact of Tracking: Evidence from a Randomized Evaluation in Kenya , 2008 .

[2]  James J. Heckman,et al.  Randomization and Social Policy Evaluation , 1991 .

[3]  H. Patrinos,et al.  Making Schools Work. , 2011 .

[4]  P. Glewwe,et al.  The impacts of school management reforms in Madagascar: do the impacts vary by teacher type? , 2011 .

[5]  P. Glewwe,et al.  Many Children Left Behind? Textbooks and Test Scores in Kenya , 2007 .

[6]  Leigh L. Linden,et al.  The Effect of Village-Based Schools: Evidence from a Randomized Controlled Trial in Afghanistan , 2012, SSRN Electronic Journal.

[7]  Cash Transfers, Behavioral Changes, and Cognitive Development in Early Childhood: Evidence from a Randomized Experiment , 2008 .

[8]  Federica Origo.  Flexible Pay, Firm Performance and the Role of Unions. New Evidence from Italy , 2009 .

[9]  Geoffrey D. Borman,et al.  Comprehensive School Reform and Achievement: A Meta-Analysis , 2003 .

[10]  Jeffrey H Silber,et al.  Optimal multivariate matching before randomization. , 2004, Biostatistics.

[11]  D. McKenzie,et al.  Returns to Capital in Microenterprises: Evidence from a Field Experiment , 2007, SSRN Electronic Journal.

[12]  M. Kremer,et al.  Incentives to Learn , 2004 .

[13]  David S. Lee.  Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects , 2009 .

[14]  C. Colclough,et al.  RECOUP Working Paper 25. Financing education in Kenya: expenditures, outcomes and the role of international aid , 2009 .

[15]  M. Kremer,et al.  Decentralization: A cautionary tale , 2003 .

[16]  Donald P. Green,et al.  Enough Already about “Black Box” Experiments: Studying Mediation Is More Difficult than Most Scholars Suppose , 2009 .

[17]  Sarah Baird,et al.  Cash or Condition? Evidence from a Cash Transfer Experiment , 2010 .

[18]  K. Lohr,et al.  Criteria for Distinguishing Effectiveness From Efficacy Trials in Systematic Reviews , 2006 .

[19]  Abhijit Banerjee,et al.  The Experimental Approach to Development Economics , 2008 .

[20]  Priyanka Pandey,et al.  Community Participation in Public Schools: The Impact of Information Campaigns in Three Indian States , 2008 .

[21]  K. Muralidharan,et al.  Contract Teachers: , 2009 .

[22]  Unions and the incidence of performance linked pay schemes in Britain , 1988 .

[23]  Leigh L. Linden,et al.  A Better Way to Teach Children to Read? Evidence from a Randomized Controlled Trial , 2010 .

[24]  S. Burgess,et al.  Incentives in Organisations : A Selective Overview of the Literature with Application to the Public Sector , 1999 .

[25]  Joshua D. Angrist,et al.  Mostly Harmless Econometrics: An Empiricist's Companion , 2008 .

[26]  A. Barr,et al.  Information and collective action in the community monitoring of schools : Field and lab experimental evidence from Uganda ∗ , 2012 .

[27]  Edward Miguel,et al.  Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities, Data User's Guide , 2014 .

[28]  John Vickers,et al.  Performance Comparisons and Dynamic Incentives , 1997, Journal of Political Economy.

[29]  James Berry.  Child Control in Education Decisions , 2015, The Journal of Human Resources.

[30]  Leigh L. Linden,et al.  Colombia - The use and misuse of computers in education : evidence from a randomized controlled trial of a language arts program , 2013 .

[31]  D. Green,et al.  Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees , 2012 .

[32]  Christina H. Paxson,et al.  Does Money Matter? The Effects of Cash Transfers on Child Health and Development in Rural Ecuador , 2007 .

[33]  Ximena López,et al.  Beyond Nintendo: design and assessment of educational video games for first and second grade students , 2003, Comput. Educ..

[34]  P. Inamdar.  Computer Skills Development by Children Using "Hole in the Wall" Facilities in Rural India. , 2004 .

[35]  Jakob Svensson,et al.  Fighting Corruption to Improve Schooling: Evidence from a Newspaper Campaign in Uganda , 2005 .

[36]  Daron Acemoglu,et al.  Theory, General Equilibrium and Political Economy in Development Economics , 2010 .

[37]  Esther Duflo,et al.  School Governance, Teacher Incentives, and Pupil-Teacher Ratios: Experimental Evidence from Kenyan Primary Schools , 2012 .

[38]  M. Kremer,et al.  School Meals, Educational Achievement, and School Competition: Evidence from a Randomized Evaluation , 2004 .

[39]  Nancy Cartwright,et al.  Evidence-Based Policy: A Practical Guide to Doing It Better , 2012 .

[40]  W. Easterly,et al.  Reinventing foreign aid , 2008 .

[41]  Stephen P. Ryan,et al.  Incentives Work: Getting Teachers to Come to School , 2012 .

[42]  Rukmini,et al.  Departnnent of Economics Working Paper Series Pitfalls of Participatory Programs : Evidence from a Randomized Evaluation in Education in India , 2011 .

[43]  Miriam Bruhn,et al.  In Pursuit of Balance: Randomization in Practice in Development Field Experiments , 2008 .

[44]  Sergio L. Schmukler,et al.  Emerging Market Instability : Do Sovereign Ratings Affect Country Risk and Stock Returns ? , 1997 .

[45]  Leigh L. Linden,et al.  How to Teach English in India: Testing the Relative Productivity of Instruction Methods within the Pratham English Language Education Program 1 , 2008 .

[46]  Angus Deaton.  Instruments, Randomization, and Learning about Development , 2010 .

[47]  Eric Zitzewitz,et al.  Retrospective vs. Prospective Analyses of School Inputs: The Case of Flip Charts in Kenya , 2004 .

[48]  S. Mullainathan,et al.  External Validity and Partner Selection Bias , 2012 .

[49]  D. Green,et al.  Modeling heterogeneous treatment effects in large-scale experiments using Bayesian Additive Regression Trees , 2010 .

[50]  Leigh L. Linden.  Complement or substitute? The effect of technology on student achievement in India , 2008 .

[51]  H. Alderman,et al.  Educational and Health Impacts of Two School Feeding Schemes: Evidence from a Randomized Trial in Rural Burkina Faso , 2009 .

[52]  K. Michaelowa,et al.  Teacher shortages, teacher contracts and their effect on education in Africa , 2007, SSRN Electronic Journal.

[53]  G. Mwabu,et al.  Why Did Abolishing Fees Not Increase Public School Enrollment in Kenya? , 2011 .

[54]  Leigh L. Linden,et al.  Computer-assisted learning: Evidence from a randomized experiment , 2011 .

[55]  Trevor Hastie,et al.  Additive Logistic Regression : a Statistical , 1998 .

[56]  J. Kmenta.  Mostly Harmless Econometrics: An Empiricist's Companion , 2010 .

[57]  F. Teal,et al.  Does performance related pay for teachers improve student performance? Some evidence from India , 2007 .

[58]  Kosuke Imai,et al.  Unpacking the Black-Box: Learning about Causal Mechanisms from Experimental and Observational Studies , 2011 .

[59]  Rakesh,et al.  Are Our Children Learning? Annual Learning Assessment ReportTanzania 2010 , 2010 .

[60]  R. Vakis,et al.  Cash Transfers, Behavioral Changes, and Cognitive Development in Early Childhood: Evidence from a Randomized Experiment , 2008 .

[61]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[62]  Alison L. Booth,et al.  Earnings, Productivity, and Performance‐Related Pay , 1999, Journal of Labor Economics.

[63]  G. Kingdon,et al.  The relative effectiveness and costs of contract and regular teachers in India , 2010 .

[64]  Karthik Muralidharan,et al.  Contract Teachers: Experimental Evidence from India , 2013 .