The contribution of process tracing to theory-based evaluations of complex aid instruments

This article focuses on methodological challenges in evaluating complex program aid interventions like budget support. We show that recent innovations in process-tracing methodology can help solve the identified challenges and increase the strength of causal inference made when using case studies in demanding settings. For the specific task of evaluating the governance effectiveness of budget support interventions, we developed a more fine-grained causal mechanism for a sub-set of the comprehensive program theory of budget support. Moreover, based on the informal use of Bayesian logic, we have elaborated on how to increase the conclusiveness of empirical evidence for one part of the theorized causal mechanism. We argue that by establishing an explicit theorized mechanism prior to empirical research and by critically judging our evidence according to an informal Bayesian logic we can remedy some of the problems at hand in much case-study research and increase the inferential leverage in complex within-case evaluation studies.

[1]  Howard White,et al.  Theory-based impact evaluation: principles and practice , 2009 .

[2]  Geske Dijkstra,et al.  Challenges in evaluating budget support and how to solve them , 2015 .

[3]  Diego Ruiz,et al.  Methodology for Evaluations of Budget Support Operations at Country Level. Issue Paper. May 2008 , 2008 .

[4]  Comparative Hypothesis Testing Via Process Tracing , 2014 .

[5]  Daniel Phillips,et al.  Addressing attribution of cause and effect in small n impact evaluations: towards an integrated framework , 2012 .

[6]  T. Falleti,et al.  Context and Causal Mechanisms in Political Analysis , 2009 .

[7]  Johansein Rutaihwa Joint Evaluation of Budget Support to Tanzania: lessons learned and recommendations for the future Final Report: Volume 1 2013(Link) , 2013 .

[8]  Sebastian Lemire,et al.  Making contribution analysis work: A practical framework for handling influencing factors and alternative explanations , 2012 .

[9]  M. Bunge Mechanism and Explanation , 1997 .

[10]  Mhairi Mackenzie,et al.  Theories of Change and Realistic Evaluation , 2007 .

[11]  Patricia J. Rogers,et al.  Purposeful Program Theory: Effective Use of Theories of Change and Logic Models , 2011 .

[12]  Julia Betts Aid Effectiveness and Governance Reforms: Applying realist principles to a complex synthesis across varied cases , 2013 .

[13]  Burkina Faso Evaluation of General Budget Support: Synthesis Report , 2006 .

[14]  Patricia J. Rogers,et al.  Program theory evaluation: Practice, promise, and problems , 2000 .

[15]  Thomas Delahais,et al.  Applying contribution analysis: Lessons from five years of practice , 2012 .

[16]  Elena Hesselmann The Limits of Control: The Accountability of Foundations and Partnerships in Global Health , 2011 .

[17]  P. Machamer,et al.  Thinking about Mechanisms , 2000, Philosophy of Science.

[18]  Niels Keijzer,et al.  Unfinished Agenda or Overtaken by Events? Applying Aid- and Development-Effectiveness Principles to Capacity Development Support , 2013 .

[19]  Peter Machamer,et al.  Activities and Causation: The Metaphysics and Epistemology of Mechanisms , 2004 .

[20]  Fadhel Kaboub Realistic Evaluation , 2004 .

[21]  F. Leeuw,et al.  Reconstructing Program Theories: Methods Available and Problems to be Solved , 2003 .

[22]  J. Gerring Causal Mechanisms: Yes, But… , 2010 .

[23]  John Mayne,et al.  Contribution analysis: Coming of age? , 2012 .

[24]  Elliot Stern,et al.  DFID Working Paper 38. Broadening the range of designs and methods for impact evaluations. , 2012 .

[25]  Rasmus Brun Pedersen,et al.  Process-Tracing Methods: Foundations and Guidelines , 2013 .

[26]  Jörg Faust,et al.  Financing poverty alleviation vs. promoting democracy? Multi-Donor Budget Support in Zambia , 2012 .

[27]  P. Hedström,et al.  Causal Mechanisms in the Social Sciences , 2010 .

[28]  Julie K. Johnson,et al.  A practical example of Contribution Analysis to a public health intervention , 2014 .

[29]  Jonathan A. Waskan,et al.  Mechanistic explanation at the limit , 2011, Synthese.

[30]  S. V. Evera Guide to methods for students of political science , 1997 .

[31]  S. Glennan Mechanisms and the nature of causation , 1996 .

[32]  R. Mayntz Mechanisms in the Analysis of Social Macro-Phenomena , 2004 .

[33]  F. Leeuw,et al.  Unpacking Black Boxes: Mechanisms and Theory Building in Evaluation , 2010 .

[34]  Elliott Sober,et al.  Absence of evidence and evidence of absence: evidential transitivity in connection with fossils, fishing, fine-tuning, and firing squads , 2009 .

[35]  H. Strang Evaluation of General Budget Support: Synthesis Report , 2006 .

[36]  Frans L. Leeuw,et al.  Linking theory-based evaluation and contribution analysis: Three problems and a few solutions , 2012 .

[37]  Andrew Bennett,et al.  Process tracing : from metaphor to analytic tool , 2014 .