Validating an evaluation checklist using a mixed method design.

When used in evaluation, checklists provide guidance for the collection of relevant evidence used to determine the merit, worth, or significance of an evaluand. The inherently systematic process found in the use of a checklist makes it highly relevant and useful for evaluative purposes. As such, the value of checklists for evaluation purposes is generally accepted. However, the methods for validating evaluation checklists are less commonly presented and lack specificity with respect to study designs and outcomes. This article addresses this deficit by presenting a case example of a mixed methods validation study applied to an evaluation checklist. The validation approach presented herein is relatively quick and was demonstrated to be feasible on a limited budget all the while providing a reasonable level of validation for the checklist. Following a brief overview of the checklist, the two-part validation study is presented followed by a discussion of the limitations of the methodology.

[1]  Lei Li,et al.  The Influence of Checklists and Roles on Software Practitioner Risk Perception and Decision-Making , 2006, Proceedings of the 39th Annual Hawaii International Conference on System Sciences (HICSS'06).

[2]  J. Owen,et al.  Evaluation and the Information Needs of Organizational Leaders , 1998 .

[3]  Daniel L. Stufflebeam,et al.  Evaluation Checklists: Practical Tools for Guiding and Judging Evaluations , 2001 .

[4]  A Vision for Evaluation , 2001 .

[5]  P. Pronovost,et al.  The checklist--a tool for error management and performance improvement. , 2006, Journal of critical care.

[6]  B. Hales,et al.  Development of medical checklists for improved quality of patient care. , 2007, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[7]  F. Schweigert The Meaning of Effectiveness in Assessing Community Initiatives , 2006 .

[8]  James W. Altschuld,et al.  Assessing the Effectiveness of Research Organizations , 1995 .

[9]  R. Yin Case Study Research: Design and Methods , 1984 .

[10]  S. Messick Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[11]  Arlene Fink,et al.  The Survey Handbook , 1995 .

[12]  Michael Scriven,et al.  THE LOGIC AND METHODOLOGY OF CHECKLISTS , 2005 .

[13]  John Rohrbaugh,et al.  A Spatial Model of Effectiveness Criteria: Towards a Competing Values Approach to Organizational Analysis , 1983 .

[14]  J. Loevinger Objective Tests as Instruments of Psychological Theory , 1957 .

[15]  Valerie J. Caracelli,et al.  Quality, Context, and Use , 2005 .

[16]  Danielle W. Mendoza,et al.  Putting Organizational Effectiveness into Practice: The Preeminent Consultation Task. , 1993 .

[17]  M. Scriven The Final Synthesis , 1994 .

[18]  Hsiu-Fang Hsieh,et al.  Three Approaches to Qualitative Content Analysis , 2005, Qualitative health research.

[19]  Nadini Persaud Conceptual and Practical Analysis of Costs and Benefits: Developing a Cost-Analysis Tool for Practical Program Evaluation , 2007 .

[20]  Chris L. S. Coryn Evaluation of Researchers and Their Research: Toward Making the Implicit Explicit , 2007 .

[21]  Sandra Mathison,et al.  Encyclopedia of Evaluation , 2004 .

[22]  Lei Chang A Psychometric Evaluation of 4-Point and 6-Point Likert-Type Scales in Relation to Reliability and Validity , 1994 .

[23]  W. Martz Evaluating Organizational Effectiveness , 2008 .

[24]  Chris L. S. Coryn,et al.  Evaluation Theory, Models, and Applications , 2007 .

[25]  D. Schroeter Sustainability evaluation: Development and validation of an evaluation checklist , 2008 .

[26]  Valerie J. Caracelli,et al.  Toward a Conceptual Framework for Mixed-Method Evaluation Designs , 1989 .

[27]  Allen C. Bluedorn,et al.  A Confirmatory Factor Analysis of the Competing Values Instrument , 1999 .

[28]  Anselm L. Strauss,et al.  Basics of qualitative research : techniques and procedures for developing grounded theory , 1998 .

[29]  D. Papadimitriou Conceptualizing effectiveness in a non‐profit organizational environment , 2007 .

[30]  N. Hoffart Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory , 2000 .

[31]  J. Rohrbaugh,et al.  A Competing Values Approach to Organizational Effectiveness , 1981 .

[32]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .