Practice makes better? Testing a model for training program evaluators in situation awareness.

Evaluation approaches should be appropriate for their contexts. Scholars and practitioners alike have widely acknowledged this view for at least four decades, and the ability to respond to context is clearly established as a core evaluator competency. Outside of evaluation, this knowledge of context, known as situation awareness, is seen as a critical feature of good decision making and as a factor that distinguishes experts from novices across a wide range of domains. Yet there are few opportunities for evaluators to explicitly build their skills in situation awareness. This study addressed that gap by examining the potential for one online training program, informed by research on deliberate practice, to accelerate evaluators' progress towards expertise in situation awareness. Built to align with evidence on developing situation awareness skills, the EvalPractice portal combined (1) an extensive 'case bank' of real-world evaluation scenarios, (2) repetitive practice that allowed novice evaluators to practice interpreting these evaluation scenarios, and (3) immediate feedback on the accuracy of these efforts based on events from the real-life version of the scenario. Findings from a small EvalPractice pilot suggest that it may be possible to improve foundational situation awareness skills using deliberate practice, but that further research is required to understand strategies for building higher-level situation awareness skills in evaluation.

[1]  Dennis K. Leedom,et al.  Training Situational Awareness Through Pattern Recognition in a Battlefield Environment , 1991 .

[2]  Thomas A. Schwandt.  `Back to the Rough Ground!' Beyond Theory to Practice in Evaluation , 2003 .

[3]  D. Rog,et al.  When background becomes foreground: Toward context‐sensitive evaluation practice , 2012 .

[4]  Louis Deslauriers,et al.  Improved Learning in a Large-Enrollment Physics Class , 2011, Science.

[5]  M R Endsley,et al.  Sources of situation awareness errors in aviation. , 1996, Aviation, space, and environmental medicine.

[6]  Christina A. Christie,et al.  Informing the Discussion on Evaluator Training , 2014 .

[7]  Christina A. Christie,et al.  What Counts as Credible Evidence in Applied Research and Evaluation Practice , 2008 .

[8]  Vimla L. Patel,et al.  Diagnostic Reasoning and Medical Expertise , 1994 .

[9]  Herbert A. Simon,et al.  Templates in Chess Memory: A Mechanism for Recalling Several Boards , 1996, Cognitive Psychology.

[10]  Cheryl A. Bolstad,et al.  Simulation-Based Situation Awareness Training , 2008 .

[11]  K. A. Ericsson,et al.  The Influence of Experience and Deliberate Practice on the Development of Superior Expert Performance , 2006 .

[12]  M. David Merrill,et al.  First principles of instruction , 2012 .

[13]  F P McKenna,et al.  Dual displays , 2000, Ergonomics.

[14]  Mica R. Endsley,et al.  Theoretical Underpinnings of Situation Awareness, A Critical Review , 2000 .

[15]  Michael Quinn Patton,et al.  Essentials of utilization-focused evaluation , 2012 .

[16]  D. Kundin,et al.  A Conceptual Framework for How Evaluators Make Everyday Practice Decisions , 2010 .

[17]  Eduardo Salas,et al.  Measurement of Team Situation Awareness in Low Experience Level Aviators , 2007 .

[18]  David Woods,et al.  Situation Awareness: A Critical But Ill-Defined Phenomenon , 1991 .

[19]  Marilynn B. Brewer,et al.  Principles and Methods of Social Research , 1986 .

[20]  R. Davies,et al.  Evaluator Training , 2014 .

[21]  Donald T. Campbell,et al.  Relabeling Internal and External Validity for Applied Social Scientists. , 1986 .

[22]  Michelle M. Robertson,et al.  Situation awareness in aircraft maintenance teams , 2000 .

[23]  Fadi Munshi,et al.  Low- versus high-fidelity simulations in teaching and assessing clinical skills , 2015 .

[24]  James G. Lollar,et al.  Salesperson knowledge distinctions and sales performance , 2014 .

[25]  Thomas A. Schwandt.  Evaluation Practice Reconsidered , 2002 .

[26]  Paul Ward,et al.  Deliberate practice and expert performance : defining the path to excellence , 2004 .

[27]  Anne T. Vo,et al.  Visualizing context through theory deconstruction: a content analysis of three bodies of evaluation theory literature. , 2013, Evaluation and program planning.

[28]  Daniela C. Schröter,et al.  Evaluator Competencies , 2008 .

[29]  B. Bläsing,et al.  Expertise affects representation structure and categorical activation of grasp postures in climbing , 2014, Front. Psychol..

[30]  M. Trevisan.  Practical Training in Evaluation: A Review of the Literature , 2004 .

[31]  Jody L. Fitzpatrick,et al.  A first step forward: Context assessment , 2012 .

[32]  M. Endsley,et al.  Building Shared Situation Awareness in Healthcare Settings , 2017 .

[33]  Marilyn Jager Adams,et al.  Situation Awareness and the Cognitive Management of Complex Systems , 1995, Hum. Factors.

[34]  Daniel L. Stufflebeam,et al.  Evaluation Checklists: Practical Tools for Guiding and Judging Evaluations , 2001 .

[35]  Jody L. Fitzpatrick,et al.  An introduction to context and its role in evaluation practice , 2012 .

[36]  Neil Charness,et al.  The role of practice and coaching in entrepreneurial skill domains: An international comparison of life-span chess skill acquisition. , 1996 .

[37]  Richard J. Schoech,et al.  EXPERIENTIAL LEARNING AND LEARNING ENVIRONMENTS: THE CASE OF ACTIVE LISTENING SKILLS , 2010 .

[38]  D. C. Howell.  Statistical Methods for Psychology , 1987 .

[39]  Earl H. McKinney,et al.  Effects of Deliberate Practice on Crisis Decision Performance , 2003, Hum. Factors.

[40]  M. G. Sherin,et al.  Using Video to Support Teachers’ Ability to Notice Classroom Interactions , 2005 .

[41]  M. Endsley.  Situation Awareness Misconceptions and Misunderstandings , 2015 .

[42]  B. Abernethy,et al.  Expert Anticipatory Skill in Striking Sports , 2012, Research quarterly for exercise and sport.

[43]  Gary Klein,et al.  Naturalistic Decision Making , 2008, Hum. Factors.

[44]  Richard H. Mogford,et al.  Mental Models and Situation Awareness in Air Traffic Control , 1997 .

[45]  Christina A. Christie,et al.  The Use of Role-Play in Teaching Evaluation , 2002 .

[46]  Eric G. Ramsey,et al.  Situational Awareness in Map Displays , 1987 .

[47]  N. M. Morris,et al.  On Looking into the Black Box: Prospects and Limits in the Search for Mental Models , 1986 .

[48]  Charmine E. J. Härtel,et al.  Defining aircrew coordination: Searching mishaps for meaning , 1989 .

[49]  Mica R. Endsley,et al.  Expertise and Situation Awareness , 2006 .

[50]  H. Simon,et al.  A simulation of memory for chess positions. , 1973 .

[51]  Henrik Artman.  Situation awareness and co-operation within and between hierarchical units in dynamic decision making , 1999 .

[52]  H. G. Klinzing,et al.  Intensity, Variety, and Accuracy in Nonverbal Cues and De-/Encoding: Two Experimental Investigations. , 2004 .

[53]  D. Cicchetti.  Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology. , 1994 .

[54]  K. Ericsson,et al.  Peak: Secrets from the New Science of Expertise , 2016 .

[55]  J. McGregor.  Context , 2018, J. Object Technol..

[56]  M. Engle,et al.  2002 Survey of Evaluation Preparation Programs in Universities , 2006 .

[57]  Patricia Sullivan,et al.  Using Interactive Video to Develop Preservice Teachers' Classroom Awareness. , 2013 .

[58]  Paul Ward,et al.  Complex Perceptual-Cognitive Expertise in a Simulated Task Environment , 2013 .

[59]  Mica R. Endsley,et al.  Toward a Theory of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.