Background: Research on evaluation use focuses on putting evaluation recommendations into practice. Prior theoretical research proposes varied frameworks for understanding the use (or lack) of program evaluation results. Purpose: Our purpose is to create and test a single, integrated framework for understanding evaluation use. This article relies on prior theoretical research regarding categories of utilization, typologies of recommendations, and factors affecting utilization to frame an empirical study of evaluation use that then tests the integrated theory. Setting: The empirical part of the article draws on post-evaluation interviews with sixteen agencies that have engaged in evaluation research. Subjects: The agencies are mostly local non-profits, but the sample also includes a state agency, a city agency, and two university-community partnerships. All agencies had undergone a program evaluation between 2003 and 2006. Intervention: Having participated in an evaluation is the main “intervention” of interest in this article, in which we consider the relationship between evaluation use theory and empirical evidence on the topic. Research Design: A qualitative approach, our research design involved examining each of the sixteen agencies within two years of their having been evaluated. Data Collection and Analysis: Data collection included structured in-person interviews with at least one key informant in each agency. In addition, a short, closed-ended survey was administered to research participants. Interview data were analyzed using content analysis of themes and grouping agencies according to their evaluation’s outcomes (favorable or not) and experiences. Survey data were analyzed with simple descriptive statistics and similarly involved a subgroup analysis, according to agencies’ reported use of evaluation. Findings: Most evaluation use is conceptual, and few agencies studied actually implemented specific evaluation recommendations. Agencies perceived recommendations as changes to rules and structure, which theory and prior research suggest suppresses use. An important human factor that influenced evaluation use was minimal post-evaluation interaction with evaluators. Some long-term influence of the evaluation is evident, but only as conceptual and not as instrumental. In fact, very little instrumental use existed in this empirical assessment. Conclusions: Evidence suggests that evaluation use in practice aligns with theory, specifically emphasizing the conceptual use dimension. The proposed integrated model of prior evaluation use theories may hold value for future theoretical and empirical work.
[1]
Marvin C. Alkin,et al.
Program Staff Perceptions of Barriers to Evaluation Implementation
,
2003
.
[2]
Ray C. Oman,et al.
Management Evaluation Studies
,
1984
.
[3]
Valerie J. Caracelli,et al.
The expanding scope of evaluation use
,
2000
.
[4]
Nancy Rhett,et al.
Building a Better Body of Evidence: New Opportunities to Strengthen Evaluation Utilization
,
2003
.
[5]
Eleanor Chelimsky,et al.
What have we learned about the politics of program evaluation
,
1987
.
[6]
David D. Marsh,et al.
Knowledge Utilization in Evaluation Efforts
,
1988
.
[7]
Hallie Preskill,et al.
Evaluation and Organizational Learning: Past, Present, and Future
,
2001
.
[8]
Eleanor Chelimsky.
The Definition and Measurement of Evaluation Quality as a Management Tool.
,
1983
.
[9]
K. Kirkhart,et al.
Reconceptualizing evaluation use: An integrated theory of influence
,
2000
.
[10]
Laura C. Leviton,et al.
Research On the Utilization of Evaluations
,
1981
.
[11]
Marvin C. Alkin.
A Guide for Evaluation Decision Makers
,
1985
.
[12]
M. F. Smith.
Evaluation utilization revisited
,
1988
.
[13]
Lyn M. Shulha,et al.
Evaluation Use: Theory, Research, and Practice since 1986
,
1997
.
[14]
Hallie Preskill,et al.
Current and Developing Conceptions of Use: Evaluation Use TIG Survey Results
,
1997
.
[15]
K. Leithwood,et al.
Current Empirical Research on Evaluation Utilization
,
1986
.
[16]
J. W. Johnston.
Increasing evaluation use: Some observations based on the results at the U.S. GAO
,
1988
.
[17]
Jennifer E. Iriti,et al.
Using Recommendations in Evaluation
,
2005
.
[18]
Hallie Preskill,et al.
The learning dimension of evaluation use
,
2000
.