Critical Friend Approach: Policy Evaluation Between Methodological Soundness, Practical Relevance, and Transparency of the Evaluation Process

1 Introduction Evaluation is marked by a great diversity of approaches and methods. In Evaluation Theory, Models, and Applications, Stufflebeam and Shinkfield (2007) distinguished no fewer than 26 often-used evaluation models (see also House, 1987). In this paper, the "critical friend" approach is presented as a new and promising approach. Despite the great variety of evaluation approaches in practice, there are elements that characterize the current use of the term "evaluation": First, evaluation is regularly understood as assessment (Stockmann, 2000, p. 12; Clarke & Dawson, 1999; Scriven, 1991). Rossi and Freeman (1993, p. 5) therefore defined evaluation as "the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs." A second, central element of evaluation underlines its basis in social sciences methodology. This is apparent in Rossi and Freeman's definition of the term as well. The third constitutive element is its practical relevance: Vedung (1999, p. 12) stated that the general practical orientation is so central that it must be a part of the definition of evaluation. In this paper, we will first present the core aspects of three typical approaches representing methodological soundness, practical relevance, or transparency of the assessment process. We will then illustrate our "critical friend" approach based on the example of the evaluation of the child care voucher project in the city of Lucerne. The critical friend approach tries to find an optimal balance between methodological soundness, practical relevance, and transparency of the assessment process. In addition, it accounts for empirical findings showing that the use of evaluations in Switzerland depends decisively on the interest of the decision makers in the results of the study (Balthasar, 2007). At the end of the paper we will discuss possibilities and limitations of the critical friend approach in policy evaluation. 2 Diversity of evaluation approaches As described above methodological soundness, practical relevance, and transparency of the assessment process can be considered as the three corners of the triangle in which most evaluation approaches can be positioned. The following sections sketch out core aspects of each of the three elements by presenting one typical approach for each element. 2.1 Methodological soundness: Donald T. Campbell In the 1960s, the criticism that common evaluation practice was not scientific led Donald T. Campbell to demand scientific soundness when planning the evaluation design. Without a doubt, Campbell and Stanley's (1963) Experimental and Quasi-Experimental Designs for Research and Campbell's (1969) paper, "Reforms as Experiments," were important foundations for the development of evaluation methodology. Campbell's ideal of experimentally designed program evaluation continues to shape evaluation science in part up to today (Stufflebeam & Shinkfield, 2007). Similar to Campbell, also Chen and Rossi (1980; Chen, 1990) worked towards scientifically based evaluation research and drew up the central ideas of theory-driven evaluation. As stated in Rossi, Freeman, and Lipsey (1999): Every program embodies a conception of the structure, functions, and procedures appropriate to attain its goals. This conception constitutes the 'logic' or plan of the program, which we have called program theory. The program theory explains why the program does what it does and provides the rationale for expecting that doing things that way will achieve the desired results. (p. 156) Methodologically oriented evaluation theories view evaluators foremost as independent specialists in methodological questions. Using clever and scientifically sound design, they provide firm answers to the question as to whether certain effects can be casually attributed to evaluated measures. …