Policy Evaluation, Randomized Controlled Trials, and External Validity – A Systematic Review

When properly implemented, Randomized Controlled Trials (RCT) can achieve a high degree of internal validity. Yet, if an RCT is to inform policy interventions that extend beyond the experimental population, it is critical to establish external validity. In this paper, we first present a theoretical framework of external validity and identify the potential hazards that compromise generalizing results beyond the studied population, namely Hawthorne effects, general equilibrium effects, specific sample problems, and special care in the provision of the randomized treatment. Second, we reviewed all RCTs published in leading economic journals between 2009 and 2014 and scrutinized the way they deal with external validity. Based on a set of objective indicators, we find that many published RCTs do not discuss hazards to external validity and do not provide the information that is necessary to assess potential problems. Apparently, external validity is not an important matter of concern during the peer review process. To conclude, we call for a more systematic approach to report the results of RCTs, including external validity dimensions.

[1]  Amanda Pallais.  Ineffiient Hiring in Entry-Level Labor Markets , 2012 .

[2]  Anna Breman,et al.  Give more tomorrow: Two field experiments on altruism and intertemporal choice , 2011 .

[3]  Dean S. Karlan,et al.  Agricultural Decisions after Relaxing Credit and Risk Constraints , 2012 .

[4]  X. Giné,et al.  Credit Market Consequences of Improved Personal Identification: Field Experimental Evidence from Malawi , 2011 .

[5]  R. Di Tella,et al.  Criminal Recidivism after Prison and Electronic Monitoring , 2009, Journal of Political Economy.

[6]  Maitreesh Ghatak,et al.  Incentives and the De Soto Effect , 2012 .

[7]  H. Allcott,et al.  Social Norms and Energy Conservation , 2011 .

[8]  Antoinette Schoar,et al.  Keeping it Simple: Financial Literacy and Rules of Thumb , 2010 .

[9]  D. McKenzie,et al.  Are Women More Credit Constrained? Experimental Evidence on Gender and Microenterprise Returns , 2008, SSRN Electronic Journal.

[10]  Emmanuel Saez,et al.  Teaching the Tax Code: Earnings Responses to an Experiment with EITC Recipients , 2009 .

[11]  Dilip Mookherjee.  New Directions in Development Economics: Theory or Empirics? - Is There Too Little Theory in Development Economics? , 2005 .

[12]  Michael K. Price,et al.  Is a Donor in Hand Better than Two in the Bush? Evidence from a Natural Field Experiment , 2008 .

[13]  Pedro C. Vicente,et al.  Votes and Violence: Evidence from a Field Experiment in Nigeria , 2014 .

[14]  Dani Rodrik,et al.  The New Development Economics: We Shall Experiment, but How Shall We Learn? , 2008 .

[15]  J. List,et al.  Testing for Altruism and Social Pressure in Charitable Giving , 2009, The quarterly journal of economics.

[16]  Jobiba Chinkhumba,et al.  The Demand for Medical Male Circumcision , 2014 .

[17]  Karen Macours,et al.  Changing Households' Investment Behaviour Through Social Interactions with Local Leaders: Evidence from a Randomised Transfer Programme , 2014 .

[18]  F. Maxwell Harper,et al.  Social Comparisons and Contributions to Online Communities: A Field Experiment on MovieLens , 2010, Computational Social Systems and the Internet.

[19]  Adrienne M. Lucas,et al.  Effects of School Quality on Student Achievement: Discontinuity Evidence from Kenya , 2014 .

[20]  Ludger Woessmann,et al.  The impact of an adult education voucher program: Evidence from a randomized field experiment , 2012 .

[21]  S. Mullainathan,et al.  Learning Through Noticing: Theory and Evidence from a Field Experiment , 2014 .

[22]  Parag A. Pathak,et al.  Explaining Charter School Effectiveness , 2011, SSRN Electronic Journal.

[23]  Michal Bauer,et al.  Behavioral Foundations of Microcredit: Experimental and Survey Evidence from Rural India , 2012, SSRN Electronic Journal.

[24]  P. Dupas,et al.  Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria Prevention Experiment , 2007 .

[25]  Jen Shang,et al.  A Field Experiment in Charitable Contribution: The Impact of Social Information on the Voluntary Provision of Public Goods , 2009 .

[26]  Clarence L. Wardell,et al.  Fundraising through online social networks: A field experiment on peer-to-peer solicitation , 2014 .

[27]  M. Finucane,et al.  Cooperation and Competition in Intergenerational Experiments in the Field and the Laboratory , 2008 .

[28]  Rajeev Dehejia,et al.  Experimental and Non-Experimental Methods in Development Economics: A Porous Dialectic , 2015 .

[29]  Dean S. Karlan,et al.  Observing Unobservables: Identifying Information Asymmetries with a Consumer Credit Field Experiment , 2005 .

[30]  Andreas Herrmann,et al.  Order in Product Customization Decisions: Evidence from Field Experiments , 2010, Journal of Political Economy.

[31]  Jakob Svensson,et al.  Power to the People: Evidence from a Randomized Field Experiment of a Community-Based Monitoring Project in Uganda , 2007 .

[32]  Aaron J. Sojourner.  Identification of Peer Effects with Missing Peer Data: Evidence from Project Star , 2013, SSRN Electronic Journal.

[33]  Nava Ashraf,et al.  Household bargaining and excess fertility: An experimental study in Zambia. , 2014 .

[34]  Erica Field,et al.  Does the Classic Microfinance Model Discourage Entrepreneurship Among the Poor? Experimental Evidence from India † , 2013 .

[35]  Jonathan Guryan,et al.  Peer Effects in the Workplace: Evidence from Random Groupings in Professional Golf Tournaments , 2007, American economic journal. Applied economics.

[36]  M. Angelucci,et al.  Indirect Effects of an Aid Program: How Do Cash Transfers Affect Ineligibles' Consumption? , 2009 .

[37]  Philip Oreopoulos,et al.  The Role of Application Assistance and Information in College Decisions: Results from the H&R Block FAFSA Experiment* , 2012 .

[38]  Erwin H. Bulte,et al.  Violent conflict and behavior: A field experiment in Burundi , 2012 .

[39]  D. McKenzie,et al.  Does Management Matter? Evidence from India , 2011 .

[40]  Explaining Charter School Effectiveness , 2013 .

[41]  Pedro C. Vicente.  Is Vote Buying Effective? Evidence from a Field Experiment in West Africa , 2014 .

[42]  Felix Oberholzer-Gee,et al.  Truth in Giving: Experimental Evidence on the Welfare Effects of Informed Giving to the Poor , 2011 .

[43]  Martin Ravallion,et al.  Fighting Poverty One Experiment at a Time: Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty : Review Essay , 2012 .

[44]  Sarah Baird,et al.  Cash or Condition? Evidence from a Cash Transfer Experiment , 2010 .

[45]  Lant Pritchett,et al.  Learning from Experiments When Context Matters , 2015 .

[46]  Macartan Humphreys,et al.  The Elements of Political Persuasion: Content, Charisma and Cue , 2014 .

[47]  Manuel Bagues,et al.  Can Gender Parity Break the Glass Ceiling?: Evidence from a Repeated Randomized Experiment , 2010 .

[48]  Richard J. Zeckhauser,et al.  Rank as an inherent incentive: Evidence from a field experiment ☆ , 2012 .

[49]  David S. Lyle.  The Effects of Peer Group Heterogeneity on the Production of Human Capital at West Point , 2009 .

[50]  Robert T. Jensen,et al.  Do Labor Market Opportunities Affect Young Women's Work and Family Decisions? Experimental Evidence from India , 2012 .

[51]  R. Vakis,et al.  Cash Transfers, Behavioral Changes, and Cognitive Development in Early Childhood: Evidence from a Randomized Experiment , 2008 .

[52]  Jonathan Zinman,et al.  Put Your Money Where Your Butt is: A Commitment Contract for Smoking Cessation , 2009 .

[53]  S. Eriksson,et al.  Do Employers Use Unemployment as a Sorting Criterion When Hiring? Evidence from a Field Experiment , 2014, SSRN Electronic Journal.

[54]  Jenny Aker.  Scaling Up What Works: Experimental Evidence on External Validity in Kenyan Education , 2013 .

[55]  Nava Ashraf,et al.  Spousal Control and Intra-household Decision Making: An Experimental Study in the Philippines , 2009 .

[56]  Esther Duflo,et al.  Peer Effects, Teacher Incentives, and the Impact of Tracking: Evidence from a Randomized Evaluation in Kenya , 2008 .

[57]  John N. Friedman,et al.  How Does Your Kindergarten Classroom Affect Your Earnings? Evidence from Project Star , 2010, The quarterly journal of economics.

[58]  R. Pande,et al.  The Economic Returns to Social Interaction: Experimental Evidence from Microfinance , 2013 .

[59]  Victor Lavy.  Performance Pay and Teachers' Effort, Productivity and Grading Ethics , 2004 .

[60]  Oriana Bandiera,et al.  No Margin, no Mission? A Field Experiment on Incentives for public service delivery , 2014 .

[61]  M. Gurgand,et al.  Private and Public Provision of Counseling to Job-Seekers: Evidence from a Large Controlled Experiment , 2014, SSRN Electronic Journal.

[62]  Erik Grönqvist,et al.  Effects of contracting out employment services: Evidence from a randomized experiment , 2013 .

[63]  Gonne Beekman,et al.  Behavioral Responses and the Impact of New Agricultural Technologies: Evidence from a Double‐Blind Field Experiment in Tanzania , 2014 .

[64]  Jonathan R.W. Temple,et al.  Aid and Conditionality , 2010 .

[65]  Eric Roetman,et al.  A can of worms? Implications of rigorous impact evaluations for development agencies , 2012 .

[66]  Steffen Huck,et al.  Matched Fundraising: Evidence from a Natural Field Experiment , 2010, SSRN Electronic Journal.

[67]  George Loewenstein,et al.  Promoting Healthy Choices: Information versus Convenience , 2010 .

[68]  Jonathan Robinson,et al.  Nudging Farmers to Use Fertilizer: Theory and Experimental Evidence from Kenya , 2009 .

[69]  J. E. West,et al.  Is Poor Fitness Contagious? Evidence from Randomly Assigned Friends , 2010 .

[70]  Rebecca Lynn Thornton,et al.  Menstruation, Sanitary Products, and School Attendance: Evidence from a Randomized Evaluation , 2011 .

[71]  Christopher Blattman,et al.  Generating Skilled Self-Employment in Developing Countries: Experimental Evidence from Uganda , 2013 .

[72]  G. Charness,et al.  Incentives to Exercise , 2008 .

[73]  Jonathan Robinson,et al.  Limited Insurance within the Household: Evidence from a Field Experiment in Kenya , 2008 .

[74]  M. Mogstad,et al.  How Financial Incentives Induce Disability Insurance Recipients to Return to Work , 2013, The American economic review.

[75]  Henrik Jacobsen Kleven,et al.  Unwilling or Unable to Cheat? Evidence From a Tax Audit Experiment in Denmark , 2011 .

[76]  P. Gertler,et al.  Investing Cash Transfers to Raise Long Term Living Standards , 2006 .

[77]  G. Harrison,et al.  Field experiments , 1924, The Journal of Agricultural Science.

[78]  Esther Duflo,et al.  Fighting Poverty One Experiment at a Time: A Review of Abhijit Banerjee and , 2016 .

[79]  Jonah E. Rockoff,et al.  Information and Employee Evaluation: Evidence from a Randomized Intervention in Public Schools , 2010 .

[80]  Junko Onishi,et al.  Should Aid Reward Performance? Evidence from a Field Experiment on Health and Education in Indonesia , 2012, American economic journal. Applied economics.

[81]  Katrina Jessoe,et al.  Knowledge is (Less) Power: Experimental Evidence from Residential Energy Use , 2012 .

[82]  Matthew J. Notowidigdo,et al.  Duration Dependence and Labor Market Conditions: Evidence from a Field Experiment* , 2013 .

[83]  Robert W. Fairlie,et al.  The Effects of Home Computers on Educational Outcomes: Evidence from a Field Experiment with Community College Students , 2012, SSRN Electronic Journal.

[84]  Edward Miguel,et al.  Spring cleaning: rural water impacts, valuation, and property rights institutions. , 2011, The quarterly journal of economics.

[85]  Jesse M. Shapiro,et al.  Can Higher Prices Stimulate Product Use? Evidence from a Field Experiment in Zambia , 2010 .

[86]  Edward Miguel,et al.  Reshaping Institutions: Evidence on Aid Impacts Using a Pre-Analysis Plan , 2011 .

[87]  Parag A. Pathak,et al.  Accountability and Flexibility in Public Schools: Evidence from Boston's Charters and Pilots , 2009 .

[88]  Steffen Q. Mueller.  Teacher experience and the class size effect - experimental evidence , 2013 .

[89]  Dean S. Karlan,et al.  Does the Media Matter? A Field Experiment Measuring the Effect of Newspapers on Voting Behavior and Political Opinions , 2006 .

[90]  Jonathan Zinman,et al.  Being surveyed can change later behavior and related parameter estimates , 2011, Proceedings of the National Academy of Sciences.

[91]  Francesco Avvisati,et al.  Getting Parents Involved: A Field Experiment in Deprived Schools , 2010 .

[92]  Flip Klijn,et al.  Constrained school choice , 2009, J. Econ. Theory.

[93]  Imran Rasul,et al.  Family Networks and School Enrolment: Evidence from a Randomized Social Experiment , 2009, SSRN Electronic Journal.

[94]  Esther Duflo,et al.  Powerful Women: Does Exposure Reduce Bias? , 2008 .

[95]  D. McKenzie,et al.  Returns to Capital in Microenterprises: Evidence from a Field Experiment , 2007, SSRN Electronic Journal.

[96]  Ragan Petrie,et al.  What Persuades Voters? A Field Experiment on Political Campaigning , 2012 .

[97]  A. de Grip,et al.  The Effects of Training on Own and Co�?Worker Productivity: Evidence from a Field Experiment , 2011, SSRN Electronic Journal.

[98]  S. Rozelle,et al.  Encouraging classroom peer interactions: Evidence from Chinese migrant schools , 2014 .

[99]  Robert T. Jensen,et al.  The (Perceived) Returns to Education and the Demand for Schooling , 2010 .

[100]  Moussa P. Blimpo.  Team Incentives for Education in Developing Countries A Randomized Field Experiment in Benin , 2014 .

[101]  Robert A. Moffitt,et al.  The Role of Randomized Field Trials in Social Science Research , 2002 .

[102]  Roland G. Fryer.  Injecting Charter School Best Practices into Traditional Public Schools: Evidence from Field Experiments , 2014 .

[103]  Olivier Armantier,et al.  Comparing Corruption in the Laboratory and in the Field in Burkina Faso and in Canada , 2013 .

[104]  Stephen P. Ryan,et al.  Incentives Work: Getting Teachers to Come to School , 2012 .

[105]  Caterina Calsamiglia,et al.  Constrained School Choice: An Experimental Study , 2009 .

[106]  Jacobus Cilliers,et al.  The white-man effect: How foreigner presence affects behavior in experiments , 2015 .

[107]  C. Meghir,et al.  Education Choices in Mexico: Using a Structural Model and a Randomized Experiment to evaluate Progresa.∗ , 2005 .

[108]  Jan Stoop,et al.  From the Lab to the Field: Cooperation among Fishermen , 2012, Journal of Political Economy.

[109]  Aprajit Mahajan,et al.  Micro-Loans, Insecticide-Treated Bednets and Malaria: Evidence from a Randomized Controlled Trial in Orissa (India) , 2011, The American economic review.

[110]  Leigh L. Linden,et al.  Improving the Design of Conditional Transfer Programs: Evidence from a Randomized Education Experiment in Colombia † , 2011 .

[111]  P. Dupas.  Short-Run Subsidies and Long-Run Adoption of New Health Products: Evidence from a Field Experiment , 2010, Econometrica : journal of the Econometric Society.

[112]  Gustavo J. Bobonis.  Is the Allocation of Resources within the Household Efficient? New Evidence from a Randomized Experiment , 2009, Journal of Political Economy.

[113]  C. Puppe,et al.  The Currency of Reciprocity - Gift-Exchange in the Workplace , 2011 .

[114]  Karthik Muralidharan,et al.  *The Impact of Diagnostic Feedback to Teachers on Student Learning: Experimental Evidence from India , 2010 .

[115]  Travis J. Lybbert,et al.  Can Mobile Phones Improve Learning? Evidence from a Field Experiment in Niger , 2012 .

[116]  E. Bulte,et al.  Corruption, investments and contributions to public goods: Experimental evidence from rural Liberia , 2014 .

[117]  Sendhil Mullainathan,et al.  What's Advertising Content Worth? Evidence from a Consumer Credit Marketing Field Experiment , 2009 .

[118]  J. List,et al.  Gender Differences in Competition: Evidence from a Matrilineal and a Patriarchal Society , 2008 .

[119]  Victor Lavy,et al.  The Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial , 2009 .

[120]  K. Telle.  Monitoring and Enforcement of Environmental Regulations: Lessons from a Natural Field Experiment in Norway , 2012 .

[121]  Jonas Hjort.  Ethnic Divisions and Production in Firms , 2013, SSRN Electronic Journal.

[122]  Christopher Woodruff,et al.  The Demand for, and Consequences of, Formalization Among Informal Firms in Sri Lanka , 2012, SSRN Electronic Journal.

[123]  Philip Oreopoulos,et al.  Incentives and Services for College Achievement: Evidence from a Randomized Trial , 2007, SSRN Electronic Journal.

[124]  Lori Beaman,et al.  Who Gets the Job Referral? Evidence from a Social Networks Experiment , 2012 .

[125]  G. Pellegrini,et al.  Do subsidies to private capital boost firms' growth? A multiple regression discontinuity design approach , 2014 .

[126]  Alois Stutzer,et al.  Active Decisions and Prosocial Behaviour: A Field Experiment on Blood Donation , 2011 .

[127]  Sendhil Mullainathan,et al.  Site Selection Bias in Program Evaluation , 2012 .

[128]  A. Banerjee,et al.  Targeting the Poor: Evidence from a Field Experiment in Indonesia , 2010, The American economic review.

[129]  Nancy Cartwright.  What are randomised controlled trials good for? , 2009 .

[130]  P. Glewwe,et al.  Many Children Left Behind? Textbooks and Test Scores in Kenya , 2007 .

[131]  Seán M. Muller,et al.  Causal Interaction and External Validity: Obstacles to the Policy Relevance of Randomized Evaluations , 2015 .

[132]  Lucas C. Coffman,et al.  The Schooling Decision: Family Preferences, Intergenerational Conflict, and Moral Hazard in the Brazilian Favelas , 2012, Journal of Political Economy.

[133]  Damon Jones.  Information, Preferences, and Public Benefit Participation: Experimental Evidence from the Advance EITC and 401(k) Savings , 2010 .

[134]  Esther Duflo,et al.  Do Labor Market Policies Have Displacement Effects? Evidence from a Clustered Randomized Experiment , 2012 .

[135]  Fredrik Carlsson,et al.  Social preferences are stable over long periods of time , 2014 .

[136]  Costas Meghir,et al.  Subsidizing Vocational Training for Disadvantaged Youth in Colombia: Evidence from a Randomized Trial , 2011 .

[137]  Roland G. Fryer,et al.  Getting Beneath the Veil of Effective Schools: Evidence from New York City , 2011 .

[138]  Costas Meghir,et al.  Risk Pooling, Risk Preferences, and Social Networks , 2012 .

[139]  Leigh L. Linden,et al.  Bringing Education to Afghan Girls: A Randomized Controlled Trial of Village-Based Schools , 2013 .

[140]  D. Deming,et al.  Better Schools, Less Crime? , 2011 .

[141]  Jonathan Robinson,et al.  Why Don&Apos;T the Poor Save More? Evidence from Health Savings Experiments , 2011, The American economic review.

[142]  Núria Rodríguez-Planas.  Longer-Term Impacts of Mentoring, Educational Services, and Learning Incentives: Evidence from a Randomized Trial in the United States , 2012 .

[143]  Elias Bareinboim,et al.  External Validity: From Do-Calculus to Transportability Across Populations , 2014, Probabilistic and Causal Inference.

[144]  Erik Snowberg,et al.  Selective Trials: A Principal-Agent Approach to Randomized Controlled Experiments , 2012 .

[145]  E. Duflo,et al.  Truth-Telling by Third-Party Auditors and the Response of Polluting Firms: Experimental Evidence from India , 2013 .

[146]  Raj Chetty,et al.  Salience and Taxation: Theory and Evidence , 2009 .

[147]  Sendhil Mullainathan,et al.  Comparison Friction: Experimental Evidence from Medicare Drug Plans , 2011, The quarterly journal of economics.

[148]  Brian McManus,et al.  The Demand for Products Linked to Public Goods: Evidence from an Online Field Experiment , 2008 .

[149]  Guido Imbens,et al.  Site Selection Bias in Program Evaluation , 2014 .

[150]  Jonathan Robinson,et al.  Experimental Evidence on the Effects of Home Computers on Academic Achievement Among Schoolchildren , 2013, SSRN Electronic Journal.

[151]  William Jack,et al.  Heckle and Chide: Results of a Randomized Road Safety Intervention in Kenya , 2009 .

[152]  John A. List,et al.  Small matches and charitable giving: Evidence from a natural field experiment , 2011 .

[153]  Esther Duflo,et al.  Peer Effects, Teacher Incentives, and the Impact of Tracking: Evidence from a Randomized Evaluation in Kenya , 2008 .

[154]  Thomas Fujiwara,et al.  Can Informed Public Deliberation Overcome Clientelism? Experimental Evidence from Benin , 2013 .

[155]  Karthik Muralidharan,et al.  School Inputs, Household Substitution, and Test Scores , 2011 .

[156]  J. Ludwig,et al.  The Effects of Housing Assistance on Labor Supply: Evidence from a Voucher Lottery , 2012 .

[157]  E. Bulte,et al.  Pseudo-Placebo Effects in Randomized Controlled Trials for Development: Evidence from a Double-Blind Field Experiment in Tanzania , 2012 .

[158]  Amanda Beatty,et al.  Improving Educational Quality Through Enhancing Community Participation: Results from a Randomized Field Experiment in Indonesia , 2011 .

[159]  G. Mwabu,et al.  Scaling Up What Works: Experimental Evidence on External Validity in Kenyan Education , 2013 .

[160]  Torsten Bumgarner.  A Can Of Worms , 2016 .

[161]  J. Svensson,et al.  POWER TO THE PEOPLE: EVIDENCE FROM A RANDOMIZED FIELD EXPERIMENT ON COMMUNITY-BASED MONITORING IN UGANDA∗ , 2008 .