Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity

When properly implemented, Randomized Controlled Trials (RCT) achieve a high degree of internal validity. Yet, if an RCT is to inform policy, it is critical to establish external validity. This paper systematically reviews all RCTs conducted in developing countries and published in leading economic journals between 2009 and 2014 with respect to how they deal with external validity. Following Duflo, Glennerster, and Kremer (2008), the hazards to external validity we scrutinize are Hawthorne effects, general equilibrium effects, specific sample problems, and special care in treatment provision. Based on a set of objective indicators, we find that the majority of published RCTs does not discuss these hazards and many do not provide the necessary information to assess potential problems. The paper calls for including external validity dimensions in a more systematic reporting on the results of RCTs. This may create incentives to avoid overgeneralizing findings and help policy makers to interpret results appropriately.

[1]  Jacobus Cilliers,et al.  The white-man effect: How foreigner presence affects behavior in experiments , 2015 .

[2]  Antoinette Schoar,et al.  Keeping it Simple: Financial Literacy and Rules of Thumb , 2010 .

[3]  Dani Rodrik,et al.  The New Development Economics: We Shall Experiment, but How Shall We Learn? , 2008 .

[4]  Lori Beaman,et al.  Who Gets the Job Referral? Evidence from a Social Networks Experiment , 2012 .

[5]  Jenny Aker.  Scaling Up What Works: Experimental Evidence on External Validity in Kenyan Education , 2013 .

[6]  Esther Duflo,et al.  Do Labor Market Policies Have Displacement Effects? Evidence from a Clustered Randomized Experiment , 2012 .

[7]  Stephen P. Ryan,et al.  Incentives Work: Getting Teachers to Come to School , 2012 .

[8]  D. Coady,et al.  Evaluating Transfer Programmes within a General Equilibrium Framework , 2004 .

[9]  Maitreesh Ghatak,et al.  Incentives and the De Soto Effect , 2012 .

[10]  P. Glewwe,et al.  Many Children Left Behind? Textbooks and Test Scores in Kenya , 2007 .

[11]  Seán M. Muller,et al.  Causal Interaction and External Validity: Obstacles to the Policy Relevance of Randomized Evaluations , 2015 .

[12]  Dean S. Karlan,et al.  Agricultural Decisions after Relaxing Credit and Risk Constraints , 2012 .

[13]  Robert T. Jensen,et al.  The (Perceived) Returns to Education and the Demand for Schooling , 2010 .

[14]  D. McKenzie,et al.  Does Management Matter? Evidence from India , 2011 .

[15]  Jonathan Zinman,et al.  Being surveyed can change later behavior and related parameter estimates , 2011, Proceedings of the National Academy of Sciences.

[16]  David R. Just,et al.  Internal and External Validity in Economics Research: Tradeoffs between Experiments, Field Experiments, Natural Experiments, and Field Data , 2009 .

[17]  Erwin H. Bulte,et al.  Violent conflict and behavior: A field experiment in Burundi , 2012 .

[18]  Costas Meghir,et al.  Subsidizing Vocational Training for Disadvantaged Youth in Colombia: Evidence from a Randomized Trial , 2011 .

[19]  Esther Duflo,et al.  Peer Effects, Teacher Incentives, and the Impact of Tracking: Evidence from a Randomized Evaluation in Kenya , 2008 .

[20]  Eric Roetman,et al.  A can of worms? Implications of rigorous impact evaluations for development agencies , 2012 .

[21]  Steven D. Levitt,et al.  Field Experiments in Economics: the Past, the Present, and the Future , 2008 .

[22]  Gary Saretsky,et al.  The OEO P.C. Experiment and the John Henry Effect. , 1972 .

[23]  Kaushik Basu,et al.  Development Economics and Method: A Quarter Century of ABCDE , 2014 .

[24]  A. Deaton,et al.  Understanding and Misunderstanding Randomized Controlled Trials , 2016, Social science & medicine.

[25]  Esther Duflo,et al.  Powerful Women: Does Exposure Reduce Bias? , 2008 .

[26]  Rajeev Dehejia,et al.  Experimental and Non-Experimental Methods in Development Economics: A Porous Dialectic , 2015 .

[27]  Brian A. Nosek,et al.  Promoting Transparency in Social Science Research , 2014, Science.

[28]  Pedro C. Vicente.  Is Vote Buying Effective? Evidence from a Field Experiment in West Africa , 2014 .

[29]  Alessio Terzi,et al.  On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics , 2014 .

[30]  Michal Bauer,et al.  Behavioral Foundations of Microcredit: Experimental and Survey Evidence from Rural India , 2012, SSRN Electronic Journal.

[31]  Adrienne M. Lucas,et al.  Effects of School Quality on Student Achievement: Discontinuity Evidence from Kenya , 2014 .

[32]  Jörg Peters,et al.  Policy Evaluation, Randomized Controlled Trials, and External Validity – A Systematic Review , 2016 .

[33]  Rachel Glennerster,et al.  The Generalizability Puzzle , 2017 .

[34]  Travis J. Lybbert,et al.  Can Mobile Phones Improve Learning? Evidence from a Field Experiment in Niger , 2012 .

[35]  P. Gertler,et al.  Investing Cash Transfers to Raise Long Term Living Standards , 2006 .

[36]  Erik Snowberg,et al.  Selective Trials: A Principal-Agent Approach to Randomized Controlled Experiments , 2010 .

[37]  E. Duflo,et al.  Truth-Telling by Third-Party Auditors and the Response of Polluting Firms: Experimental Evidence from India , 2013 .

[38]  Catherine P. Bradshaw,et al.  The use of propensity scores to assess the generalizability of results from randomized trials , 2011, Journal of the Royal Statistical Society. Series A,.

[39]  R. Vakis,et al.  Cash Transfers, Behavioral Changes, and Cognitive Development in Early Childhood: Evidence from a Randomized Experiment , 2008 .

[40]  Amanda Beatty,et al.  Improving Educational Quality Through Enhancing Community Participation: Results from a Randomized Field Experiment in Indonesia , 2011 .

[41]  Dean S. Karlan,et al.  Teaching Entrepreneurship: Impact of Business Training on Microfinance Clients and Institutions , 2006, Review of Economics and Statistics.

[42]  Christopher Blattman,et al.  Generating Skilled Self-Employment in Developing Countries: Experimental Evidence from Uganda , 2013 .

[43]  Aprajit Mahajan,et al.  Micro-Loans, Insecticide-Treated Bednets and Malaria: Evidence from a Randomized Controlled Trial in Orissa (India) , 2011, The American economic review.

[44]  David I. Levine,et al.  Using Unobtrusive Sensors to Measure and Minimize Hawthorne Effects: Evidence from Cookstoves , 2017 .

[45]  M. Kremer,et al.  The Two-Margin Problem in Insurance Markets , 2004, Review of Economics and Statistics.

[46]  Nava Ashraf,et al.  Spousal Control and Intra-household Decision Making: An Experimental Study in the Philippines , 2009 .

[47]  Kaushik Basu,et al.  Randomisation, Causality and the Role of Reasoned Intuition , 2014 .

[48]  Costas Meghir,et al.  Risk Pooling, Risk Preferences, and Social Networks , 2012 .

[49]  Rebecca Lynn Thornton,et al.  Menstruation, Sanitary Products, and School Attendance: Evidence from a Randomized Evaluation , 2011 .

[50]  Robert A. Moffitt,et al.  The Role of Randomized Field Trials in Social Science Research , 2002 .

[51]  R. Di Tella,et al.  Criminal Recidivism after Prison and Electronic Monitoring , 2009, Journal of Political Economy.

[52]  D. Elbourne,et al.  Systematic review of the Hawthorne effect: New concepts are needed to study research participation effects☆ , 2014, Journal of clinical epidemiology.

[53]  Jakob Svensson,et al.  Power to the People: Evidence from a Randomized Field Experiment of a Community-Based Monitoring Project in Uganda , 2007 .

[54]  Amanda E. Kowalski.  How to Examine External Validity within an Experiment , 2018, Journal of economics & management strategy.

[55]  J. List,et al.  Gender Differences in Competition: Evidence from a Matrilineal and a Patriarchal Society , 2008 .

[56]  Jonas Hjort.  Ethnic Divisions and Production in Firms , 2013, SSRN Electronic Journal.

[57]  D. McKenzie,et al.  Are Women More Credit Constrained? Experimental Evidence on Gender and Microenterprise Returns , 2008, SSRN Electronic Journal.

[58]  Edward Miguel,et al.  Reshaping Institutions: Evidence on Aid Impacts Using a Pre-Analysis Plan , 2011 .

[59]  Robert T. Jensen,et al.  Do Labor Market Opportunities Affect Young Women's Work and Family Decisions? Experimental Evidence from India , 2012 .

[60]  Moussa P. Blimpo.  Team Incentives for Education in Developing Countries A Randomized Field Experiment in Benin , 2014 .

[61]  J. Behrman,et al.  Timing and Duration of Exposure in Evaluations of Social Programs , 2008 .

[62]  Nava Ashraf,et al.  Household bargaining and excess fertility: An experimental study in Zambia. , 2014 .

[63]  Olivier Armantier,et al.  Comparing Corruption in the Laboratory and in the Field in Burkina Faso and in Canada , 2013 .

[64]  A. Banerjee,et al.  Targeting the Poor: Evidence from a Field Experiment in Indonesia , 2010, The American economic review.

[65]  R. Pande,et al.  The Economic Returns to Social Interaction: Experimental Evidence from Microfinance , 2013 .

[66]  Christopher Woodruff,et al.  What are We Learning from Business Training and Entrepreneurship Evaluations Around the Developing World? , 2012, SSRN Electronic Journal.

[67]  Marshall Burke,et al.  Selling low and buying high: An arbitrage puzzle in Kenyan villages , 2014 .

[68]  Anna Popova,et al.  Insecure Property Rights and the Missing Middle , 2015 .

[69]  Harini Kannan,et al.  From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application , 2016 .

[70]  X. Giné,et al.  Credit Market Consequences of Improved Personal Identification: Field Experimental Evidence from Malawi , 2011 .

[71]  Anna Popova,et al.  What Really Works to Improve Learning in Developing Countries? An Analysis of Divergent Findings in Systematic Reviews , 2015 .

[72]  G. Mwabu,et al.  Scaling Up What Works: Experimental Evidence on External Validity in Kenyan Education , 2013 .

[73]  A. Deaton.  Instruments of Development: Randomization in the Tropics, and the Search for the Elusive Keys to Economic Development , 2009 .

[74]  Jonathan Robinson,et al.  Nudging Farmers to Use Fertilizer: Theory and Experimental Evidence from Kenya , 2009 .

[75]  Junko Onishi,et al.  Should Aid Reward Performance? Evidence from a Field Experiment on Health and Education in Indonesia , 2012, American economic journal. Applied economics.

[76]  P. Dupas.  Short-Run Subsidies and Long-Run Adoption of New Health Products: Evidence from a Field Experiment , 2010, Econometrica : journal of the Econometric Society.

[77]  Christopher Woodruff,et al.  The Demand for, and Consequences of, Formalization Among Informal Firms in Sri Lanka , 2012, SSRN Electronic Journal.

[78]  Jesse M. Shapiro,et al.  Can Higher Prices Stimulate Product Use? Evidence from a Field Experiment in Zambia , 2010 .

[79]  Leigh L. Linden,et al.  Improving the Design of Conditional Transfer Programs: Evidence from a Randomized Education Experiment in Colombia † , 2011 .

[80]  Jessica Cohen,et al.  Free Distribution or Cost-Sharing? Evidence from a Malaria Prevention Experiment , 2008 .

[81]  Karthik Muralidharan,et al.  *The Impact of Diagnostic Feedback to Teachers on Student Learning: Experimental Evidence from India , 2010 .

[82]  Jobiba Chinkhumba,et al.  The Demand for Medical Male Circumcision , 2014 .

[83]  D. McKenzie,et al.  Returns to Capital in Microenterprises: Evidence from a Field Experiment , 2007, SSRN Electronic Journal.

[84]  Shah Ebrahim,et al.  The Impact of Health Insurance Schemes for the Informal Sector in Low- and Middle-Income Countries: A Systematic Review , 2013 .

[85]  Amanda Beatty,et al.  Improving Educational Quality Through Enhancing Community Participation: Results from a Randomized Field Experiment in Indonesia , 2011 .

[86]  Edward Miguel,et al.  Spring cleaning: rural water impacts, valuation, and property rights institutions. , 2011, The quarterly journal of economics.

[87]  Lant Pritchett,et al.  Learning from Experiments When Context Matters , 2015 .

[88]  Esther Duflo,et al.  Fighting Poverty One Experiment at a Time: A Review of Abhijit Banerjee and , 2016 .

[89]  Jonathan Zinman,et al.  Put Your Money Where Your Butt is: A Commitment Contract for Smoking Cessation , 2009 .

[90]  Martin Ravallion,et al.  Fighting Poverty One Experiment at a Time: Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty : Review Essay , 2012 .

[91]  D. Moher,et al.  CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials , 2010, BMJ : British Medical Journal.

[92]  Lucas C. Coffman,et al.  The Schooling Decision: Family Preferences, Intergenerational Conflict, and Moral Hazard in the Brazilian Favelas , 2012, Journal of Political Economy.

[93]  Nancy Cartwright.  What are randomised controlled trials good for? , 2009 .

[94]  Georg Kirchsteiger,et al.  Assignment Procedure Biases in Randomised Policy Experiments , 2017 .

[95]  Dean S. Karlan,et al.  Observing Unobservables: Identifying Information Asymmetries with a Consumer Credit Field Experiment , 2005 .

[96]  Michael Woolcock,et al.  Using case studies to explore the external validity of ‘complex’ development interventions , 2013 .

[97]  Elias Bareinboim,et al.  External Validity: From Do-Calculus to Transportability Across Populations , 2014, Probabilistic and Causal Inference.

[98]  Karthik Muralidharan,et al.  General Equilibrium Effects of (Improving) Public Employment Programs: Experimental Evidence from India , 2017, Econometrica.

[99]  Erica Field,et al.  Does the Classic Microfinance Model Discourage Entrepreneurship Among the Poor? Experimental Evidence from India † , 2013 .

[100]  Jonathan Robinson,et al.  Limited Insurance within the Household: Evidence from a Field Experiment in Kenya , 2008 .

[101]  Karthik Muralidharan,et al.  School Inputs, Household Substitution, and Test Scores , 2011 .

[102]  Gonne Beekman,et al.  Behavioral Responses and the Impact of New Agricultural Technologies: Evidence from a Double‐Blind Field Experiment in Tanzania , 2014 .

[103]  Jonathan R.W. Temple,et al.  Aid and Conditionality , 2010 .

[104]  Guido Imbens,et al.  Site Selection Bias in Program Evaluation , 2014 .

[105]  Karthik Muralidharan,et al.  Preliminary and Incomplete: Please do not cite or circulate without authors’ permission The Aggregate Effect of School Choice: Evidence from a two-stage experiment in India , 2012 .

[106]  Leigh L. Linden,et al.  Bringing Education to Afghan Girls: A Randomized Controlled Trial of Village-Based Schools , 2013 .

[107]  Pedro C. Vicente,et al.  Votes and Violence: Evidence from a Field Experiment in Nigeria , 2014 .

[108]  S. Mullainathan,et al.  Learning Through Noticing: Theory and Evidence from a Field Experiment , 2014 .

[109]  Jonathan Robinson,et al.  Why Don&Apos;T the Poor Save More? Evidence from Health Savings Experiments , 2011, The American economic review.

[110]  C. Meghir,et al.  Education Choices in Mexico: Using a Structural Model and a Randomized Experiment to evaluate Progresa.∗ , 2005 .

[111]  Sarah Baird,et al.  Cash or Condition? Evidence from a Cash Transfer Experiment , 2010 .

[112]  A. Mosqueira,et al.  Technical Proposal for Replication of "Power to the People: Evidence from a Randomized Field Experiment on Community-Based Monitoring in Uganda" , 2014 .

[113]  Sendhil Mullainathan,et al.  What's Advertising Content Worth? Evidence from a Consumer Credit Marketing Field Experiment , 2009 .

[114]  Nolan H. Miller,et al.  Do Consumer Price Subsidies Really Improve Nutrition? , 2008, Review of Economics and Statistics.

[115]  Thomas Fujiwara,et al.  Can Informed Public Deliberation Overcome Clientelism? Experimental Evidence from Benin , 2013 .

[116]  D. Moher,et al.  CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials , 2010, Journal of pharmacology & pharmacotherapeutics.

[117]  Achyuta Adhvaryu,et al.  Learning, Misallocation, and Technology Adoption: Evidence from New Malaria Therapy in Tanzania. , 2014, The Review of economic studies.

[118]  Karen Macours,et al.  Changing Households' Investment Behaviour Through Social Interactions with Local Leaders: Evidence from a Randomised Transfer Programme , 2014 .

[119]  Angus Deaton.  Instruments, Randomization, and Learning about Development , 2010 .

[120]  Anna Popova,et al.  Cash Transfers and Temptation Goods , 2017, Economic Development and Cultural Change.