Timing and Duration of Exposure in Evaluations of Social Programs

Impact evaluations aim to measure the outcomes that can be attributed to a specific policy or intervention. Although there have been excellent reviews of the different methods that an evaluator can choose in order to estimate impact, there has not been sufficient attention given to questions related to timing: How long after a program has begun should one wait before evaluating it? How long should treatment groups be exposed to a program before they can be expected to benefit from it? Are there important time patterns in a program's impact? Many impact evaluations assume that interventions occur at specified launch dates and produce equal and constant changes in conditions among eligible beneficiary groups; but there are many reasons why this generally is not the case. This paper examines the evaluation issues related to timing and discusses the sources of variation in the duration of exposure within programs and their implications for impact estimates. It reviews the evidence from careful evaluations of programs (with a focus on developing countries) on the ways that duration affects impacts.

[1]  Shaohua Chen,et al.  Are there lasting impacts of aid to poor areas , 2009 .

[2]  Norbert Schady,et al.  Cash Transfers, Conditions, and School Enrollment in Ecuador , 2008 .

[3]  K. Leonard,et al.  Is patient satisfaction sensitive to changes in the quality of care? An exploitation of the Hawthorne effect. , 2008, Journal of health economics.

[4]  Rukmini,et al.  Departnnent of Economics Working Paper Series Pitfalls of Participatory Programs : Evidence from a Randomized Evaluation in Education in India , 2011 .

[5]  Dominique P. van de Walle,et al.  Rural Roads and Poor Area Development in Vietnam , 2007 .

[6]  R. Martorell,et al.  Child development in developing countries 3 Strategies to avoid the loss of developmental potential in more than 200 million children in the developing world , 2007 .

[7]  P. Glewwe,et al.  Developmental potential in the first 5 years for children in developing countries , 2007, The Lancet.

[8]  A. Ezeh,et al.  Family planning: the unfinished agenda , 2006, The Lancet.

[9]  E. Galasso,et al.  Learning Through Monitoring: Lessons from a Large-Scale Nutrition Program in Madagascar , 2006 .

[10]  Petra E. Todd,et al.  Assessing the Impact of a School Subsidy Program in Mexico: Using a Social Experiment to Validate a Dynamic Behavioral Model of Child Schooling and Fertility. , 2006, The American economic review.

[11]  E. King,et al.  Early Childhood Development Through an Integrated Program: Evidence from the Philippines , 2006 .

[12]  Aimee Chin,et al.  Can redistributing teachers across schools raise educational attainment? Evidence from Operation Blackboard in India , 2005 .

[13]  S. Walker,et al.  Effects of early childhood psychosocial stimulation and nutritional supplementation on cognition and education in growth-stunted Jamaican children: prospective cohort study , 2005, The Lancet.

[14]  Petra E. Todd,et al.  Progressing through PROGRESA: An Impact Assessment of a School Subsidy Experiment in Rural Mexico , 2005, Economic Development and Cultural Change.

[15]  Duncan C. Thomas,et al.  Can expanding access to basic healthcare improve children's health status? Lessons from Indonesia's ‘midwife in the village’ programme , 2004, Population studies.

[16]  M. Ravallion,et al.  What Can Ex-Participants Reveal about a Program’s Impact? , 2005, The Journal of Human Resources.

[17]  A. Mills,et al.  The economics of social marketing: the case of mosquito nets in Tanzania. , 2005, Social science & medicine.

[18]  Edward Miguel,et al.  Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities, Guide to Replication of Miguel and Kremer (2004) , 2014 .

[19]  J. Angrist,et al.  Long-Term Consequences of Secondary School Vouchers: Evidence from Administrative Records in Colombia , 2004 .

[20]  Alan B. Krueger,et al.  Putting Computerized Instruction to the Test: A Randomized Evaluation of a "Scientifically-Based" Reading Program , 2004 .

[21]  Kaivan Munshi.  Social learning in a heterogeneous population: technology diffusion in the Indian Green Revolution , 2004 .

[22]  Petra E. Todd,et al.  Evaluating Preschool Programs When Length of Exposure to the Program Varies: A Nonparametric Approach , 2000, Review of Economics and Statistics.

[23]  Duncan C. Thomas,et al.  Can Expanding Access to Basic Health Care Improve Children's Health Status? Lessons from Indonesia's "Midwife in the Village" Program (Basic Health Care and Children's Health Status) , 2004 .

[24]  P. Gertler.  Do Conditional Cash Transfers Improve Child Health? Evidence from PROGRESA’s Control Randomized Experiment. , 2004, The American economic review.

[25]  John Hoddinott,et al.  Long-term consequences of early childhood malnutrition , 2006 .

[26]  Thomas F. Crossley,et al.  Econometrics for Evaluations: An Introduction to Recent Developments , 2003 .

[27]  Petra E. Todd,et al.  Using a Social Experiment to Validate a Dynamic Behavioral Model of Child Schooling and Fertility: Assessing the Impact of a School Subsidy Program in Mexico , 2003 .

[28]  Laura B. Rawlings,et al.  Evaluating the Impact of Conditional Cash Transfer Programs: Lessons from Latin America , 2003 .

[29]  Esther Duflo,et al.  Scaling Up and Evaluation , 2003 .

[30]  John Hoddinott,et al.  Programme Evaluation with Unobserved Heterogeneity and Selective Implementation: The Mexican Progresa Impact on Child Nutrition , 2005 .

[31]  M. Rowland,et al.  Prevention of malaria in Afghanistan through social marketing of insecticide‐treated nets: evaluation of coverage and effectiveness by cross‐sectional surveys and passive surveillance , 2002, Tropical medicine & international health : TM & IH.

[32]  Laura B. Rawlings,et al.  An impact evaluation of education, health, and water supply investments by the Bolivian social investment fund , 2002 .

[33]  M. Ravallion.  Are the Poor Protected from Budget Cuts? Evidence for Argentina , 2002 .

[34]  J. Hoddinott,et al.  Child Growth in the Time of Drought , 2001 .

[35]  T. Schultz.  SCHOOL SUBSIDIES FOR THE POOR: EVALUATING THE MEXICAN PROGRESA POVERTY PROGRAM , 2012 .

[36]  M. Ravallion.  The Mystery of the Vanishing Benefits: An Introduction to Impact Evaluation , 2001 .

[37]  Victor Lavy,et al.  Does Teacher Training Affect Pupil Learning? Evidence from Matched Comparisons in Jerusalem Public Schools , 1998, Journal of Labor Economics.

[38]  Duncan C. Thomas,et al.  Longer Term Effects of Head Start , 2000 .

[39]  E. Duflo.  Schooling and Labor Market Consequences of School Construction in Indonesia: Evidence from an Unusual Policy Experiment , 2000 .

[40]  Bruce Fuller,et al.  Nicaragua's Experiment to Decentralize Schools: Contrasting Views of Parents, Teachers, and Directors , 1999, Comparative Education Review.

[41]  Peter F. Orazem,et al.  Central Mandates and Local Incentives: The Colombia Education Voucher Program , 1999 .

[42]  J. Heckman,et al.  The Economics and Econometrics of Active Labor Market Programs , 1999 .

[43]  Y. Sawada,et al.  Do Community-Managed Schools Work? An Evaluation of El Salvador's EDUCO Program. Working Paper Series on Impact Evaluation of Education Reforms. Paper No. 8. , 1998 .

[44]  Elizabeth King,et al.  What's Decentralization Got To Do With Learning? The Case of Nicaragua's School Autonomy Reform , 1998 .

[45]  Charles F. Manski,et al.  Learning about Treatment Effects from Experiments with Random Assignment of Treatments , 1996 .

[46]  M. Rosenzweig,et al.  Learning by Doing and Learning from Others: Human Capital and Technical Change in Agriculture , 1995, Journal of Political Economy.

[47]  James J. Heckman,et al.  Assessing the Case for Social Experiments , 1995 .

[48]  G. Rodríguez,et al.  The Mass Media and Family Planning in Kenya , 1995 .

[49]  J. Grossman,et al.  Evaluating social policies : principles and U.S. experience , 1994 .

[50]  M. Rosenzweig,et al.  The Determinants and Consequences of the Placement of Government Programs in Indonesia , 1993 .

[51]  D. Mihaljek,et al.  Project evaluation and uncertainty in practice : a statistical analysis of rate-of-return divergences of 1,015 World Bank projects , 1992 .

[52]  M. Rosenzweig,et al.  Evaluating the Effects of Optimally Distributed Public Programs: ChildHealth and Family Planning Interventions , 1986 .