Reported Influence of Evaluation Data on Decision Makers’ Actions

Using a set of scenarios derived from actual evaluation studies, this simulation study examines the reported influence of evaluation information on decision makers’ potential actions. Each scenario described a context where one of three types of evaluation information (large-scale study data, case study data, or anecdotal accounts) is presented and a specific decision needs to be made. Participants were asked to indicate which type of data presented would influence their decision making. Results from 131 participants indicate that participants were influenced by all types of information, yet large-scale and case study data are more influential relative to anecdotal accounts; certain types of evaluation data are more influential among certain groups of decision makers; and choosing to use one type of evaluation data over the other two depends on the independent influence of other types of evaluation data on the decision maker, as well as prior beliefs about program efficacy.

[1]  Alvin C. Burns,et al.  A Framework for Assessing the Competencies Reflected in Simulation Performance , 1999 .

[2]  R. Hogarth,et al.  Decision Making under Ambiguity , 1986 .

[3]  John R. Dickinson,et al.  An Empirical Investigation of the Internal Validity of A Marketing Simulation Game John , 1990 .

[4]  K. Glaister,et al.  Breaking the frame: an analysis of strategic cognition and decision making under uncertainty , 1999 .

[5]  Dianna L. Newman,et al.  The Credibility of a Local Educational Program Evaluation Report: Author Source and Client Audience Characteristics , 1978 .

[6]  E A Leventhal,et al.  Impact of symptoms and aging attribution on emotions and coping. , 1987, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.

[7]  C. Weiss.  Using social research in public policy making , 1979 .

[8]  Michael Quinn Patton,et al.  Utilization-Focused Evaluation , 1979 .

[9]  Gary T. Henry,et al.  Why not use? , 2000 .

[10]  Hugh M. Cannon,et al.  Constructs of Simulation Evaluation , 2002 .

[11]  Gordon C. Rausser,et al.  On the Limitations of Simulation in Model Evaluation and Decision Analysis , 1975 .

[12]  K. Kirkhart,et al.  Reconceptualizing evaluation use: An integrated theory of influence , 2000 .

[13]  Christina A. Christie,et al.  The 2004 Claremont Debate: Lipsey vs. Scriven Determining Causality in Program Evaluation & Applied Research: Should Experimental Evidence Be the Gold Standard? , 2004 .

[14]  Melvin M. Mark,et al.  Beyond Use: Understanding Evaluation’s Influence on Attitudes and Actions , 2003 .

[15]  Dianna L. Newman,et al.  Evaluator Credibility as a Function of Report Style , 1978 .

[16]  Erin Murphy-Graham,et al.  An Alternate Route to Policy Influence , 2005 .

[17]  John C. Ory,et al.  Faculty perceptions of the quality and usefulness of three types of evaluative information , 1981 .

[18]  Robert C. Ford,et al.  Ethical decision making: A review of the empirical literature , 1994 .

[19]  Harold F. O'Neil,et al.  Workforce Readiness: Competencies and Assessment. , 1997 .

[20]  Amy L. Pablo.  Determinants of acquisition integration level: A decision-making perspective. , 1994 .

[21]  Melvin M. Mark,et al.  The Mechanisms and Outcomes of Evaluation Influence , 2004 .

[22]  Michael Quinn Patton,et al.  In Search of Impact: An Analysis of the Utilization of Federal Health Evaluation Research. , 1975 .

[23]  Jane L. Winer,et al.  The Relative Efficacy of Simulation Experiments. , 1983 .

[24]  Dianna L. Newman,et al.  Does the Superintendent's Opinion Affect School Boards' Evaluation Information Needs? , 1985 .

[25]  I. Rosenstock,et al.  Social Learning Theory and the Health Belief Model , 1988, Health education quarterly.

[26]  W. Edwards.  The theory of decision making. , 1954, Psychological bulletin.

[27]  Richard R. Hake,et al.  Possible Palliatives for the Paralyzing Pre/Post Paranoia that Plagues Some PEP’s , 2007, Journal of MultiDisciplinary Evaluation.

[28]  Richard J. Shavelson,et al.  Scientific Culture and Educational Research , 2002 .

[29]  A. K. Basu.  A Theory of Decision-Making , 1973, The Journal of Sociology & Social Welfare.

[30]  Dianna L. Newman,et al.  An Investigation of the Effect of Different Data Presentation Formats and Order of Arguments in a Simulated Adversary Evaluation , 1982 .

[31]  Dianna L. Newman,et al.  Studying Evaluation Utilization Through Simulations , 1982 .

[32]  Dale E. Berger,et al.  Applied Psychology : New Frontiers and Rewarding Careers , 2006 .

[33]  Stephen M. Alessi,et al.  Fidelity in the design of instructional simulations , 1988 .