Timely evaluation in international development

ABSTRACT Impact and process evaluations are increasingly used in international development; however they are generally retrospective in outlook. A more timely approach to evaluation aims to identify necessary, feasible and effective changes during a programme or intervention’s lifetime. This paper aims to identify, categorise, describe and critically appraise methods to support more timely evaluation in international development. Potential methods were identified through scoping seminar, public symposium, targeted review of the literature, and the authors’ own experiences and opinions. Findings from the different data sources were reviewed collectively by the author group and triangulated to develop an analytical framework. We identified four purposes of timely evaluation for international development, and critiqued the use of approaches against four dimensions of timeliness and flexibility. Whilst we found significant interest in more timely approaches to evaluation in international development, there was a dearth of published empirical evidence upon which to base strong recommendations. There is significant potential for timely evaluation to improve international development outcomes. New approaches to mixing and adapting existing methods, together with new technologies offer increased potential. Research is needed to provide an empirical evidence base upon which to further develop the application, across sectors and contexts, of timely evaluation in international development.

[1]  Sendhil Mullainathan,et al.  What's Advertising Content Worth? Evidence from a Consumer Credit Marketing Field Experiment , 2009 .

[2]  A. Gasparrini,et al.  Interrupted time series regression for the evaluation of public health interventions: a tutorial , 2016, International journal of epidemiology.

[3]  D. Siewiorek,et al.  Ecological Momentary Assessment in Behavioral Research: Addressing Technological and Human Participant Challenges , 2017, Journal of medical Internet research.

[4]  Tanya Marchant,et al.  Identifying implementation bottlenecks for maternal and newborn health interventions in rural districts of the United Republic of Tanzania , 2015, Bulletin of the World Health Organization.

[5]  James Beebe,et al.  Rapid Assessment Process: An Introduction , 2001 .

[6]  Trudie Lang Adaptive Trial Design: Could We Use This Approach to Improve Clinical Trials in the Field of Global Health? , 2011, The American journal of tropical medicine and hygiene.

[7]  C Barnett,et al.  Process Tracing: The Potential and Pitfalls for Impact Evaluation in International Development. Summary of a Workshop held on 7 May 2014 , 2014 .

[8]  O. Doumbo,et al.  Prevention of Malaria in Pregnancy with Intermittent Preventive Treatment and Insecticide Treated Nets in Mali: A Quantitative Health Systems Effectiveness Analysis , 2013, PloS one.

[9]  Lenore Manderson,et al.  Can rapid anthropological procedures be applied to tropical diseases , 1992 .

[10]  D. Moher,et al.  A scoping review of rapid review methods , 2015, BMC Medicine.

[11]  M. Hamel,et al.  Effectiveness of the delivery of interventions to prevent malaria in pregnancy in Kenya , 2016, Malaria Journal.

[12]  P. Hildebrand Combining disciplines in rapid appraisal: The Sondeo approach , 1981 .

[13]  Heather Kane,et al.  Using qualitative comparative analysis to understand and quantify translation and implementation , 2014, Translational behavioral medicine.

[14]  Donald A. Dinero Use and Misuse , 2011 .

[15]  A. Manzano,et al.  The craft of interviewing in realist evaluation , 2016 .

[16]  Stephen B. Fawcett,et al.  Rapid Assessment Procedures: A Review and Critique , 1997 .

[17]  Alexander C. Wagenaar,et al.  The Value of Interrupted Time-Series Experiments for Community Intervention Research , 2000, Prevention Science.

[18]  Margareth Crisóstomo Portela,et al.  How to study improvement interventions: a brief overview of possible study types , 2015, BMJ Quality & Safety.

[19]  G. R. Alexander The Adjacent Possible. , 2019, Nursing education perspectives.

[20]  Holger J Schünemann,et al.  Reviews: Rapid! Rapid! Rapid! …and systematic , 2015, Systematic Reviews.

[21]  T. Hallett,et al.  Providing a conceptual framework for HIV prevention cascades and assessing feasibility of empirical measurement with data from east Zimbabwe: a case study , 2016, The lancet. HIV.

[22]  J. Benneyan,et al.  Statistical process control as a tool for research and healthcare improvement , 2003, Quality & safety in health care.

[23]  Lynn Westbrook,et al.  Utilization-focused evaluation , 1998 .

[24]  Jack Bowden,et al.  Response‐adaptive designs for binary responses: How to offer patient benefit while being robust to time trends? , 2017, Pharmaceutical statistics.

[25]  Daniel Phillips,et al.  Addressing attribution of cause and effect in small n impact evaluations: towards an integrated framework , 2012 .

[26]  Marcel Tanner,et al.  The relevance of rapid assessment to health research and interventions , 1992 .

[27]  B. Befani,et al.  Process Tracing and Contribution Analysis: A Combined Approach to Generative Causal Inference for Impact Evaluation , 2014 .

[28]  J. Copestake,et al.  Managing relationships in qualitative impact evaluation of international development: QuIP choreography as a case study , 2018 .

[29]  C. Coffey,et al.  Adaptive trial designs: a review of barriers and opportunities , 2012, Trials.

[30]  Frank Bretz,et al.  Adaptive Trial Designs , 2011 .

[31]  Sudirman Nasir,et al.  Use of most significant change (MSC) technique to evaluate health promotion training of maternal community health workers in Cianjur district, Indonesia. , 2018, Evaluation and program planning.

[32]  Candice Bowman,et al.  The role of formative evaluation in implementation research and the QUERI experience , 2006, Journal of General Internal Medicine.

[33]  T. Tanahashi Health service coverage and its evaluation. , 1978, Bulletin of the World Health Organization.

[34]  C. Karema,et al.  The relative roles of ANC and EPI in the continuous distribution of LLINs: a qualitative study in four countries , 2017, Health policy and planning.

[35]  Anne Lene Haukanes Hopstad,et al.  Key Design Considerations , 2016 .

[36]  B. Freidlin,et al.  Adaptive Clinical Trials: Advantages and Disadvantages of Various Adaptive Design Elements , 2017, Journal of the National Cancer Institute.

[37]  A. Kesselheim,et al.  Adaptive design clinical trials: a review of the literature and ClinicalTrials.gov , 2018, BMJ Open.

[38]  M. Dixon-Woods,et al.  The problem with root cause analysis , 2016, BMJ Quality & Safety.

[39]  Giel Ton,et al.  The mixing of methods: A three-step process for improving rigour in impact evaluations , 2012 .

[40]  R. Ratnayake,et al.  Effects of a community scorecard on improving the local health system in Eastern Democratic Republic of Congo: qualitative evidence using the most significant change technique , 2015, Conflict and Health.

[41]  R. Mahajan,et al.  Adaptive design clinical trials: Methodology, challenges and prospect , 2010, Indian journal of pharmacology.

[42]  Kristian Thorlund,et al.  Key design considerations for adaptive clinical trials: a primer for clinicians , 2018, British Medical Journal.

[43]  Barbara Befani,et al.  Between complexity and generalization: Addressing evaluation challenges with QCA , 2013 .

[44]  Thibaut Jombart,et al.  Key data for outbreak evaluation: building on the Ebola experience , 2017, Philosophical Transactions of the Royal Society B: Biological Sciences.

[45]  I. Speizer,et al.  An implementation science protocol of the Women’s Health CoOp in healthcare settings in Cape Town, South Africa: A stepped-wedge design , 2017, BMC Women's Health.

[46]  Matt Andrews,et al.  Explaining Positive Deviance in Public Sector Reforms in Development , 2015 .

[47]  Michael Woolcock,et al.  Toward a plurality of methods in project evaluation: a contextualised approach to understanding impact trajectories and efficacy , 2009 .

[48]  Amy Javernick-Will,et al.  Use and misuse of qualitative comparative analysis , 2011 .

[49]  Between the clinic and the community: Temporality and patterns of ART adherence in the Western Cape Province South Africa. , 2015 .

[50]  Lorenzo Trippa,et al.  A Bayesian response-adaptive trial in tuberculosis: The endTB trial , 2017, Clinical trials.

[51]  J. Copestake,et al.  Assessing Rural Transformations: Piloting a Qualitative Impact Protocol in Malawi and Ethiopia , 2015 .

[52]  Pieterbas Lalleman,et al.  Realist Evaluation , 2019, TVZ - Verpleegkunde in praktijk en wetenschap.

[53]  Deepak L. Bhatt,et al.  Adaptive Designs for Clinical Trials. , 2016, The New England journal of medicine.

[54]  Niall Keleher,et al.  Innovations for Poverty Action , 2007 .

[55]  A. O’Cathain,et al.  Process evaluation of complex interventions: Medical Research Council guidance , 2015, BMJ : British Medical Journal.

[56]  Rasmus Brun Pedersen,et al.  Process-Tracing Methods: Foundations and Guidelines , 2013 .

[57]  Donna Ciliska,et al.  Expediting systematic reviews: methods and implications of rapid reviews , 2010, Implementation science : IS.

[58]  D. Ladner STRATEGY TESTING: AN INNOVATIVE APPROACH TO MONITORING HIGHLY FLEXIBLE AID PROGRAMS , 2015 .

[59]  Valéry Ridde,et al.  The concept of mechanism from a realist approach: a scoping review to facilitate its operationalization in public health program evaluation , 2015, Implementation Science.

[60]  Terry Smutylo,et al.  Outcome Mapping: A method for tracking behavioural changes in development programs , 2005 .

[61]  James Lopez Bernal,et al.  The use of controls in interrupted time series studies of public health interventions. , 2018, International journal of epidemiology.

[62]  J. Copestake Credible impact evaluation in complex contexts: Confirmatory and exploratory approaches , 2014 .

[63]  S. Shiffman,et al.  Ecological momentary assessment. , 2008, Annual review of clinical psychology.

[64]  J. Busza,et al.  Learning from returnee Ethiopian migrant domestic workers: a qualitative assessment to reduce the risk of human trafficking , 2017, Globalization and Health.

[65]  Michael Bamberger,et al.  INTRODUCTION TO MIXED METHODS IN IMPACT EVALUATION , 2012 .

[66]  Evangelos Kontopantelis,et al.  Regression based quasi-experimental approach when randomisation is not an option: interrupted time series analysis , 2015, BMJ : British Medical Journal.

[67]  James Mahoney,et al.  A tale of two cultures: qualitative and quantitative research in the social sciences and Social science concepts: a user's guide and Explaining war and peace: case studies and necessary condition counterfactuals and Case studies, causal mechanisms, and selecting cases , 2015 .

[68]  O’Donoghue,et al.  Getting to the Top of Mind : How Reminders Increase Saving , 2014 .

[69]  Rick Davies An evolutionary approach to facilitating organisational learning: an experiment by the Christian Commission for Development in Bangladesh , 1998 .

[70]  Fred Carden,et al.  Outcome mapping : building learning and reflection into development programs [Arabic version] , 2002 .

[71]  E. Aagaard,et al.  Evaluating the Impact of the Medical Education Partnership Initiative at the University of Zimbabwe College of Health Sciences Using the Most Significant Change Technique. , 2017, Academic medicine : journal of the Association of American Medical Colleges.

[72]  Tanya M. Swart,et al.  Rapid Assessment Procedures , 2003 .

[73]  Mae Keary,et al.  The Science of Evaluation: A Realist Manifesto , 2014, Online Inf. Rev..

[74]  Claes Wohlin,et al.  Guidelines for snowballing in systematic literature studies and a replication in software engineering , 2014, EASE '14.

[75]  A. Lépine,et al.  Investigating interventions to increase uptake of HIV testing and linkage into care or prevention for male partners of pregnant women in antenatal clinics in Blantyre, Malawi: study protocol for a cluster randomised trial , 2017, Trials.

[76]  Maria J Grant,et al.  A typology of reviews: an analysis of 14 review types and associated methodologies. , 2009, Health information and libraries journal.

[77]  Robert Klemmensen A Tale of Two Cultures – Qualitative and Quantitative: Research in Social Sciences by Gary Goertz and James Mahony. Princeton NJ: Princeton University Press, 2012. 248pp., £19.95, ISBN 978 0691149714 , 2014 .