Program theory evaluation: Logic analysis.

Program theory evaluation, which has grown in use over the past 10 years, assesses whether a program is designed in such a way that it can achieve its intended outcomes. This article describes a particular type of program theory evaluation-logic analysis-that allows us to test the plausibility of a program's theory using scientific knowledge. Logic analysis is useful for improving the intervention or finding alternatives for achieving intended outcomes; it influences the choice of type of evaluation to conduct and strengthens the validity of subsequent evaluations. The objective of this article is to present the methodological principles and the roots of this type of program theory evaluation. We illustrate two types of logic analysis with two actual evaluation cases. There are very few published examples of program theory evaluation. This article will provide evaluators with both theoretical and practical information to help them in conceptualizing their evaluations.

[1]  M. Patton.  Utilization-focused evaluation: The new century text , 1997 .

[2]  B E Puetz,et al.  Evaluation research. , 1982, Journal of continuing education in nursing.

[3]  Mark W. Lipsey,et al.  Theory as method: Small theories of treatments , 1993 .

[4]  Astrid Brousselle,et al.  Using Logic Analysis to Evaluate Knowledge Transfer Initiatives , 2009, Evaluation.

[5]  Carol H. Weiss,et al.  Which links in which theories shall we evaluate? , 2000 .

[6]  Michael Quinn Patton,et al.  The Evaluator's Responsibility for Utilization. , 1988 .

[7]  J Frenk,et al.  Balancing relevance and excellence: organizational responses to link research with decision making. , 1992, Social science & medicine.

[8]  Huey-Tsyh Chen,et al.  Practical program evaluation : assessing and improving planning, implementation, and effectiveness , 2005 .

[9]  Astrid Brousselle,et al.  Use of health care services by patients with co-occurring severe mental illness and substance use disorders. , 2008, Mental health and substance use : dual diagnosis.

[10]  John W. Kingdon.  Agendas, alternatives, and public policies , 1984 .

[11]  A. G. Jordan.  The Commercial lobbyists : politics for profit in Britain , 1991 .

[12]  Huey-Tsyh Chen.  The Roots of Theory-Driven Evaluation: Current Views and Origins , 2004 .

[13]  Carol Hirschon Weiss,et al.  Theory-Based Evaluation in Practice , 2000, Evaluation review.

[14]  J Lomas,et al.  Review of the Multi-Hospital Arrangements Literature: Benefits, Disadvantages and Lessons for Implementation , 1995, Healthcare management forum.

[15]  Carol H. Weiss.  Evaluation Research: Methods of Assessing Program Effectiveness , 1972 .

[16]  Daniel Reinharz,et al.  Involving decision-makers in producing research syntheses: the case of the research collective on primary healthcare in quebec. , 2007, Healthcare policy = Politiques de sante.

[17]  P Howden-Chapman,et al.  Translating research findings into health policy. , 1996, Social science & medicine.

[18]  André-Pierre Contandriopoulos,et al.  3. Modéliser les interventions , 2011 .

[19]  Louise Potvin,et al.  Health promotion evaluation practices in the Americas : values and research , 2008 .

[20]  T. Greenhalgh,et al.  Realist review - a new method of systematic review designed for complex policy interventions , 2005, Journal of health services research & policy.

[21]  Fadhel Kaboub.  Realistic Evaluation , 2004 .

[22]  Huey-Tsyh Chen,et al.  Issues in constructing program theory , 1990 .

[23]  André-Pierre Contandriopoulos,et al.  6. L’analyse logique , 2011 .

[24]  L. Milbrath,et al.  LOBBYING AS A COMMUNICATION PROCESS , 1960 .

[25]  Astrid Brousselle,et al.  Evaluating interventions aimed at promoting information utilization in organizations and systems. , 2008, Healthcare policy = Politiques de sante.

[26]  Carol H. Weiss,et al.  Theory‐based evaluation: Past, present, and future , 1997 .

[27]  J. M. Beyer,et al.  The Utilization Process: A Conceptual Framework and Synthesis of Empirical Findings. , 1982 .

[28]  Peter H. Rossi,et al.  The theory-driven approach to validity , 1987 .

[29]  Stephen M. Shortell,et al.  Remaking Health Care in America: Building Organized Delivery Systems , 1996 .

[30]  Jules M. Marquart.  A pattern‐matching approach to link program theory and evaluation data , 1990 .

[31]  E. Davidson,et al.  Ascertaining causality in theory‐based evaluation , 2000 .

[32]  Astrid Brousselle,et al.  Integrating services for patients with mental and substance use disorders: What matters? , 2010, Health care management review.

[33]  Stephen Page,et al.  "Virtual" Health Care Organizations and the Challenges of Improving Quality , 2003, Health care management review.

[34]  Alan I Green,et al.  Improving the Care of Individuals with Schizophrenia and Substance Use Disorders: Consensus Recommendations , 2005, Journal of psychiatric practice.

[35]  Frank R. Baumgartner,et al.  Basic Interests: The Importance of Groups in Politics and in Political Science , 1998 .

[36]  Jean Hartley,et al.  Case study research , 2004 .

[37]  R. Yin.  Case Study Research: Design and Methods , 1984 .

[38]  Jonathan Lomas,et al.  Using research to inform healthcare managers' and policy makers' questions: from summative to interpretive synthesis. , 2005, Healthcare policy = Politiques de sante.

[39]  David Lazer,et al.  Friends, Brokers, and Transitivity: Who Informs Whom in Washington Politics? , 2004, The Journal of Politics.

[40]  Lee Sechrest,et al.  Understanding Causes and Generalizing About Them , 1993 .

[41]  Mary Ann Scheirer,et al.  Program theory and implementation theory: Implications for evaluators , 1987 .

[42]  André-Pierre Contandriopoulos,et al.  Une expérience d'intégration des soins dans une zone rurale: les enjeux de la mise en œuvre , 2001 .

[43]  Daniel Reinharz,et al.  The research collective: a tool for producing timely, context-linked research syntheses. , 2006, Healthcare policy = Politiques de sante.

[44]  Mark W. Lipsey,et al.  Evaluation: A Systematic Approach , 1979 .

[45]  Beth C. Weitzman,et al.  Integrating a Comparison Group Design into a Theory of Change Evaluation: The Case of the Urban Health Initiative: , 2002 .

[46]  Pascale Lehoux,et al.  Analyse de l'implantation d'un réseau interhospitalier de soins pédiatriques : Le Réseau Mère-Enfant , 2001 .

[47]  J. Denis,et al.  Convergent evolution: The academic and policy roots of collaborative research , 2003, Journal of health services research & policy.

[48]  Geneviève Mercille,et al.  A Realist Approach to the Systematic Review , 2008 .

[49]  Melvin M. Mark,et al.  From program theory to tests of program theory , 1990 .

[50]  Itai Sened,et al.  The Role of Lobbyists: Entrepreneurs with Two Audiences , 1993 .

[51]  David Austen-Smith,et al.  Information and influence: Lobbying for agendas and votes , 1993 .

[52]  E. Suchman.  Evaluative Research: Principles and Practice in Public Service and Social Action Progr , 1968 .

[53]  Christina A. Christie,et al.  Evaluation Roots: Tracing Theorists′ Views and Influences , 2004 .

[54]  DOUGLAS A. CONRAD,et al.  Integrated Health Systems: Promise and Performance , 1996, Frontiers of health services management.

[55]  Jody L. Fitzpatrick.  Dialogue with Stewart Donaldson , 2002 .

[56]  Patricia J. Rogers,et al.  Theory-Based Evaluation: Reflections Ten Years On , 2007 .

[57]  Sandra Mathison,et al.  Encyclopedia of Evaluation , 2004 .

[58]  Paul Davis,et al.  The Limits of Realist Evaluation , 2005 .

[59]  Mark W. Lipsey,et al.  Driving toward theory in program evaluation: More models to choose from , 1989 .

[60]  Stewart I. Donaldson.  Theory-Driven Program Evaluation in the New Millennium , 2003 .

[61]  Astrid Brousselle,et al.  Key enhancing factors for integrating services for patients with mental and substance use disorders , 2010 .

[62]  T D'Aunno.  Managing the care of health and the cure of disease: arguments for the importance of integration. , 2001, Health care management review.

[63]  Marie-Josée Fleury,et al.  « Émergence des réseaux intégrés de services comme modèle d'organisation et de transformation du système sociosanitaire » , 2002 .

[64]  Patricia J. Rogers,et al.  Program theory evaluation: Practice, promise, and problems , 2000 .

[65]  Astrid Brousselle,et al.  Beyond the limitations of best practices: how logic analysis helped reinterpret dual diagnosis guidelines. , 2007, Evaluation and program planning.

[66]  Louise Potvin,et al.  Health Promotion Evaluation Practices in the Americas , 2009 .

[67]  Carol H. Weiss,et al.  Theory‐based evaluation: Reflections ten years on: Theory‐based evaluation: Past, present, and future , 2007 .

[68]  Ralph Renger,et al.  A Three-Step Approach to Teaching Logic Models , 2002 .

[69]  François Champagne,et al.  L'évaluation dans le domaine de la santé : concepts et méthodes. , 2000 .

[70]  Kendon J. Conrad,et al.  Measuring and testing program philosophy , 1987 .

[71]  Paul A. Lamarche,et al.  L'intégration des services: enjeux structurels et organisationnels ou humains et cliniques? , 2001 .

[72]  C. Weiss.  How Can Theory-Based Evaluation Make Greater Headway? , 1997 .

[73]  Lester W. Milbrath.  The Washington lobbyists , 1963 .

[74]  C. Lindblom.  THE SCIENCE OF MUDDLING THROUGH , 1959 .

[75]  Karyen Chu,et al.  Outcome Report: State Health Policy Information: What Worked? , 1997 .

[76]  Robert E. Drake,et al.  The Challenge of Implementing and Sustaining Integrated Dual Disorders Treatment Programs , 2002, Community Mental Health Journal.

[77]  Ralph Renger,et al.  From theory to practice: Lessons learned in the application of the ATM approach to developing logic models , 2006 .

[78]  M. Trevisan.  Evaluability Assessment From 1986 to 2006 , 2007 .

[79]  Leonard Bickman,et al.  Advances in program theory , 1990 .

[80]  Jr. William R. Shadish.  Program micro‐ and macrotheories: A guide for social change , 1987 .

[81]  E. Brunswik,et al.  The psychology of Egon Brunswik , 1966 .

[82]  Geraldine Fitzpatrick,et al.  Muddling Through , 1997, Cosmopolitan Conservatisms.

[83]  C. Adair,et al.  Knowledge transfer and exchange: review and synthesis of the literature. , 2007, The Milbank quarterly.

[84]  M. Albanese,et al.  Integrated Treatment for Dual Disorders: A Guide to Effective Practice , 2004 .

[85]  Richard Reading.  Realist review to understand the efficacy of school feeding programmes , 2008 .

[86]  Timothy A. Hacsi.  Using program theory to replicate successful programs , 2000 .

[87]  E Shapiro,et al.  From research to policy: what have we learned? , 1999, Medical care.

[88]  D. Lazer,et al.  The Strength of Strong Ties , 2003 .

[89]  M. Patton,et al.  Evaluation: A Systematic Approach , 1980 .

[90]  Huey-tsyh Chen.  Theory-driven evaluations , 1990 .

[91]  Stewart I. Donaldson,et al.  Program Theory-Driven Evaluation Science: Strategies and Applications , 2007 .

[92]  Ralph Renger,et al.  How Using a Logic Model Refined Our Program to Ensure Success , 2009, Health promotion practice.

[93]  Leila J. Rupp.  Editor's Note , 2002 .

[94]  François Champagne,et al.  Introduction: Towards a Broader Understanding of the Use of Knowledge and Evidence in Health Care , 2004 .

[95]  Thomas D. Cook,et al.  The false choice between theory‐based evaluation and experimentation , 2000 .

[96]  Patricia J. Rogers,et al.  Program theory in evaluation : challenges and opportunities , 2000 .

[97]  Nick L. Smith,et al.  Using path analysis to develop and evaluate program theory and impact , 1990 .

[98]  David Lazer,et al.  The Strength of Weak Ties in Lobbying Networks , 1998 .

[99]  P H Feldman,et al.  State health policy information: what worked? , 1997, Health affairs.

[100]  M. C. Reed.  Utilization-focused Evaluation: The New Century Text (3rd ed.) , 1999 .

[101]  T. Tripodi,et al.  Evaluating social programs and problems: visions for the new millennium , 2003 .

[102]  Daniel Gacoin.  La mise en œuvre , 2010 .

[103]  J. Berry,et al.  The Interest Group Society , 1984 .

[104]  L. Bickman.  Using program theory in evaluation , 1987 .

[105]  Henry Mintzberg,et al.  Managing the care of health and the cure of disease--Part II: Integration. , 2001, Health care management review.

[106]  J. Lavis,et al.  How can research organizations more effectively transfer research knowledge to decision makers? , 2003, The Milbank quarterly.