Impact evaluation of natural resource management research programs: a broader view

Experimental impact evaluation methods have recently emerged as a dominant force within the development effectiveness movement. Although these methods have improved understanding of what works, their “gold standard” status threatens to exclude a large body of alternative evidence. This paper evaluates the impact of CARE’s SHOUHARDO project in Bangladesh, which employed a rights-based, livelihoods approach. Using a mixed-methods protocol, we find plausible evidence that the project led to an extraordinarily large reduction in child malnutrition. While offering valuable policy lessons, we illustrate how rigorous evaluation can be undertaken even without the randomization and control groups required by the experimental methods.

[1]  Howard White,et al.  Theory-based impact evaluation: principles and practice , 2009 .

[2]  Olivia Freeman,et al.  Talking points personal outcomes approach: practical guide. , 2012 .

[3]  Robert Warner,et al.  Mama Lus Frut scheme: an assessment of poverty reduction , 2002 .

[4]  R. McLeod Improved Methods for the Diagnosis and Control of Bluetongue in Small Ruminants in Asia and the Epidemiology and Control of Bovine Ephemeral Fever in China , 2003 .

[5]  Consultative Group on International Agricultural Research SCIENCE COUNCIL Standing Panel on Impact Assessment (SPIA) Natural Resources Management Research Impacts: Evidence from the CGIAR , 2006 .

[6]  David Pearce,et al.  Salinity reduction in tannery effluents in India and Australia , 2009 .

[7]  P. S. George,et al.  Increased efficiency of straw utilisation by cattle and buffalo , 1998 .

[8]  Elliot Stern,et al.  DFID Working Paper 38. Broadening the range of designs and methods for impact evaluations. , 2012 .

[9]  A. Janvry,et al.  Recent Advances in Impact Analysis Methods for Ex-post Impact Assessments of Agricultural Technology: Options for the CGIAR , 2010 .

[10]  Astrid Brousselle,et al.  Program theory evaluation: Logic analysis. , 2011, Evaluation and program planning.

[11]  D. Pearce,et al.  Saving a Staple Crop: Impact of Biological Control of the Banana Skipper on Poverty Reduction in Papua New Guinea , 2003 .

[12]  J. Mahoney Toward a Unified Theory of Causality , 2008 .

[13]  D. Pearce IDENTIFYING THE SEX PHEROMONE OF THE SUGARCANE BORER MOTH , 2005 .

[14]  M. V. Bueren Acacia Hybrids in Vietnam , 2004 .

[15]  Craig Valters,et al.  Understanding Theory of Change in International Development , 2012 .

[16]  Patricia J. Rogers,et al.  Theory-Based Evaluation: Reflections Ten Years On , 2007 .

[17]  John Mayne,et al.  Contribution analysis: An approach to exploring cause and effect , 2008 .

[18]  Jitendra,et al.  RICE-WHEAT CROPPING SYSTEMS IN INDIA , 2002 .

[19]  David Pearce,et al.  Benefits to Australia of selected CABI products , 2006 .

[20]  J. Turbin Policy borrowing: lessons from European attempts to transfer training practices , 2001 .

[21]  M. Patton Advocacy Impact Evaluation , 2008, Journal of MultiDisciplinary Evaluation.

[22]  Nicoletta Stame,et al.  Theory-Based Evaluation and Types of Complexity , 2004 .

[23]  Hayden Fisher,et al.  Improved Australian tree species for Vietnam , 2007 .

[24]  B. A. Collins,et al.  Fruit fly in Malaysia and Thailand 1985-1993 , 1998 .

[25]  M. Bantilan,et al.  Spillover Impacts of Agricultural Research: A Review of Studies , 2006 .

[26]  S. Al-Athel,et al.  Report of the World Commission on Environment and Development: "Our Common Future" , 1987 .

[27]  J. Mullen IMPACT ASSESSMENT OF ACIAR-FUNDED PROJECTS ON GRAIN-MARKET REFORM IN CHINA , 2004 .

[28]  Chris L. S. Coryn,et al.  A Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009 , 2011 .

[29]  Patricia J. Rogers,et al.  Purposeful Program Theory: Effective Use of Theories of Change and Logic Models , 2011 .

[30]  D. Vere RESEARCH INTO CONSERVATION TILLAGE FOR DRYLAND CROPPING IN AUSTRALIA AND CHINA , 2005 .

[31]  B. Shiferaw,et al.  Methods for Assessing the Impacts of Natural Resource Management Research A Summary of the Proceedings of the ICRISAT-NCAP/ICAR International Workshop , 2003 .

[32]  I. Vogel,et al.  Review of the use of 'Theory of Change' in international development. Review Report. , 2012 .

[33]  David Pearce,et al.  Benefits to Australia from ACIAR-funded Research , 2006 .

[34]  Mywish K. Maredia,et al.  Improving the proof: Evolution of and emerging trends in impact assessment methods and approaches in agricultural development , 2009 .

[35]  R. Lindner Impact Assessment of Research on the Biology and Management of Coconut Crabs on Vanuatu , 2004 .

[36]  David N. Harris Water management in public irrigation schemes in Vietnam , 2006 .

[37]  R. Prasad,et al.  Rice–Wheat Cropping Systems , 2005 .

[38]  Kevin Chadwick,et al.  Impact assessment of capacity building and training: assessment framework and two case studies , 2007 .

[39]  Ross McLeod,et al.  Control of Footrot in Small Ruminants of Nepal , 2001 .

[40]  P. Sheehan,et al.  The Contribution of Agricultural Growth to Poverty Reduction , 2012 .

[41]  Robert K. Lindner,et al.  Review of the Returns to ACIAR's Bilateral R&D Investments , 2005 .

[42]  D. Pearce,et al.  Growing trees on salt-affected land , 2007 .

[43]  Nancy Cartwright,et al.  Evidence-Based Policy: A Practical Guide to Doing It Better , 2012 .

[44]  Hayden Fisher,et al.  ACIAR's activities in Africa: a review , 2012 .

[45]  D. Pearce Measuring the poverty impact of ACIAR projects: a broad framework , 2002 .

[46]  H. Gregersen,et al.  Enhancing ex post impact assessment of agricultural research: the CGIAR experience , 2008 .

[47]  Mhairi Mackenzie,et al.  Theories of Change and Realistic Evaluation , 2007 .

[48]  R. McLeod MANAGEMENT OF FRUIT FLIES IN THE PACIFIC , 2005 .

[49]  H. Huyse,et al.  Evaluating NGO-capacity development interventions: Enhancing frameworks, fitting the (Belgian) context , 2012 .

[50]  A. Müller The Cement of the Universe: A Study of Causation , 1977 .

[51]  J. Turnbull Development of sustainable forestry plantations in China: a review , 2007 .

[52]  D. Pearce REVIEW OF ACIAR'S RESEARCH ON AGRICULTURAL POLICY , 2005 .

[53]  J. Mullen,et al.  Management of internal parasites in goats in the Philippines , 2008 .

[54]  M. Harding,et al.  Analysis of ACIAR's returns on investment: appropriateness, effi ciency and eff ectiveness , 2009 .

[55]  David N. Harris Extending low-cost fish farming in Thailand: an ACIAR-World Vision collaborative program , 2010 .

[56]  Frans L. Leeuw,et al.  Linking theory-based evaluation and contribution analysis: Three problems and a few solutions , 2012 .

[57]  C. Tisdell,et al.  Breeding and feeding pigs in Australia and Vietnam , 2001 .

[58]  C. Leeuwis,et al.  Learning about Advocacy , 2008 .

[59]  A review and impact assessment of ACIAR's fruit-fly research partnerships 1984-2007 , 2008 .

[60]  Astrid Brousselle,et al.  Using Logic Analysis to Evaluate Knowledge Transfer Initiatives , 2009, Evaluation.

[61]  R. McLeod Improved Methods in Diagnosis, Epidemiology, and Information Management of Foot-and-Mouth Disease in Southeast Asia , 2003 .

[62]  John P. Brennan,et al.  Genetics of, and Breeding for, Rust Resistance in Wheat in India and Pakistan , 2004 .

[63]  Tingsong Jiang,et al.  SHELF-LIFE EXTENSION OF LEAFY VEGETABLES: EVALUATING THE IMPACTS , 2005 .

[64]  R. Mackay,et al.  Evaluating Organizational Capacity Development , 2002, Canadian Journal of Program Evaluation.

[65]  Michael Woolcock,et al.  Toward a plurality of methods in project evaluation: a contextualised approach to understanding impact trajectories and efficacy , 2009 .

[66]  David N. Harris Water and Nitrogen Management in Wheat-Maize Production on the North China Plain , 2004 .

[67]  R. Lindner Impacts of Mud Crab Hatchery Technology in Vietnam , 2005 .

[68]  Carol H. Weiss,et al.  Theory‐based evaluation: Past, present, and future , 1997 .

[69]  D. Pearce,et al.  Zero Tillage for Weed Control in India: the Contribution to Poverty Alleviation , 2006 .

[70]  David P. Vincent,et al.  Controlling Phalaris Minor in the Indian Rice-Wheat Belt , 2002 .

[71]  Amy R. Poteete,et al.  Working Together: Collective Action, the Commons, and Multiple Methods in Practice , 2010 .

[72]  Boru Douthwaite,et al.  Participatory Impact Pathways Analysis: a practical method for project planning and evaluation , 2010 .

[73]  D. Horton Planning, Implementing, and Evaluating Capacity Development 1 , 2002 .

[74]  F. Khan,et al.  Admissible Evidence in the Court of Development Evaluation? The Impact of CARE’s SHOUHARDO Project on Child Stunting in Bangladesh , 2011 .