Plan for the Assessment and Evaluation of Individual and Team Proficiencies Developed by the DARWARS Environments

Abstract : The goal of this final report is to produce a first cut for a feasible, valid, and cost-sensitive evaluation plan to permit the development and application of metrics to assess the impact of participating in a DARWARS environment. DARWARS is a DARPA program whose goals have evolved over time due to changing requirements and technology opportunities. The original goal as reflected in the RFP was to transform military training by providing continuously available, on- demand, mission-level training for all forces at all echelons. DARPA proposed to create an environment where there will be a set of continuously available virtual training wars in which any unit or individual can participate via personal computer-based systems that will teach and exercise users in the cognitive and affective aspects of their warfare areas via electronic wargames. The current goal as reflected in documentation at the 2004 Interservice/Industry Training, Simulation and Education Conference is "to accelerate the development and deployment of the next generation of experiential training systems. These low-cost, web-centric, simulation-based systems take advantage of the ubiquity of the PC and of new technologies, including multiplayer games, virtual worlds, intelligent agents, and on-line communities. DARWARS training systems offer engaging practice environments for individuals and teams with on-target feedback for each student".

[1]  James E. Driskell,et al.  Games, Motivation, and Learning: A Research and Practice Model , 2002 .

[2]  Margaret Morrison,et al.  A Media Buying Simulation Game Using the Internet , 1998 .

[3]  Sharon Tkacz,et al.  Learning Map Interpretation: Skill Acquisition and Underlying Abilities , 1998 .

[4]  C. S. Green,et al.  Action video game modifies visual selective attention , 2003, Nature.

[5]  Giuseppe Riva,et al.  Communication and Cooperation in Networked Environments: An Experimental Analysis , 2001, Cyberpsychology Behav. Soc. Netw..

[6]  S. Toulmin.  The uses of argument , 1960 .

[7]  R. Hembree.  Correlates, Causes, Effects, and Treatment of Test Anxiety , 1988 .

[8]  Lise Busk Kofoed,et al.  Reflection in Learning Processes Through Simulation/Gaming , 1998 .

[9]  Jon-Chao Hong,et al.  A study on thinking strategy between experts and novices of computer games , 2003, Comput. Hum. Behav..

[10]  Jeffrey A. Jordan,et al.  A Dyadic Protocol for Training Complex Skills , 1992 .

[11]  Richard E. Clark,et al.  Blueprints for complex learning: The 4C/ID-model , 2002 .

[12]  Brent D. Ruben,et al.  Simulations, Games, and Experience-Based Learning: The Quest for a New Paradigm for Teaching and Learning , 1999 .

[13]  John Sweller,et al.  Cognitive Load During Problem Solving: Effects on Learning , 1988, Cogn. Sci..

[14]  Yoav Yair,et al.  3D-Virtual Reality in Science Education: An Implication for Astronomy Teaching , 2001 .

[15]  Izak Benbasat,et al.  An Experimental Investigation of Interface Design Alternatives: Icon vs. Text and Direct Manipulation vs. Menus , 1993, Int. J. Man Mach. Stud..

[16]  F. Paas,et al.  Cognitive Architecture and Instructional Design , 1998 .

[17]  E. Day,et al.  Knowledge structures and the acquisition of a complex skill. , 2001, The Journal of applied psychology.

[18]  Gregory K. W. K. Chung,et al.  Computer-based performance assessments: a solution to the narrow measurement and reporting of problem-solving , 1999 .

[19]  Richard E. Mayer,et al.  Multimedia learning in an interactive self-explaining environment: What works in the design of agent , 2003 .

[20]  Eva L. Baker,et al.  Measuring problem solving in computer environments: current and future states , 2002, Comput. Hum. Behav..

[21]  Dylan M. Jones,et al.  The effects of maps on navigation and search strategies in very-large-scale virtual environments , 1999 .

[22]  Robert D. Macredie,et al.  Games and the Design of Human‐Computer Interfaces , 1994 .

[23]  W. Montague,et al.  Aptitude, Learning, and Instruction , 1980 .

[24]  Paul J. Sticha,et al.  Evaluation of a Spatial Data Management System for Basic Skills Education , 1986 .

[25]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[26]  J. Sweller.  Evolution of human cognitive architecture , 2003 .

[27]  J. Sweller.  COGNITIVE LOAD THEORY, LEARNING DIFFICULTY, AND INSTRUCTIONAL DESIGN , 1994 .

[28]  Tom Cobb,et al.  Cognitive efficiency: Toward a revised theory of media , 1997 .

[29]  Sivasailam Thiagarajan,et al.  The Myths and Realities of Simulations in Performance Technology. , 1998 .

[30]  Patricia D. Mautone,et al.  Social cues in multimedia learning: Role of speaker's voice. , 2003 .

[31]  J. Harackiewicz,et al.  Motivating the Academically Unmotivated: A Critical Issue for the 21st Century , 2000 .

[32]  Patricia A. Chalmers,et al.  The role of cognitive theory in human-computer interface , 2003, Comput. Hum. Behav..

[33]  R. Mayer.  Should there be a three-strikes rule against pure discovery learning? The case for guided methods of instruction. , 2004, The American psychologist.

[34]  Harold F. O'Neil.  Perspectives on computer-based assessment of problem solving , 2002, Comput. Hum. Behav..

[35]  T. Goetz,et al.  Academic Emotions in Students' Self-Regulated Learning and Achievement: A Program of Qualitative and Quantitative Research , 2002 .

[36]  Timothy L. Seifert,et al.  Understanding student motivation , 2004 .

[37]  Richard E. Mayer,et al.  A taxonomy for computer-based assessment of problem solving , 2002, Comput. Hum. Behav..

[38]  P. Kirschner,et al.  Taking the Load Off a Learner's Mind: Instructional Design for Complex Learning , 2003 .

[39]  Katrina E. Ricci,et al.  Do Computer-Based Games Facilitate Knowledge Acquisition and Retention? , 1996 .

[40]  Patricia D. Mautone,et al.  Pictorial aids for learning by doing in a multimedia geology simulation game. , 2002 .

[41]  Kristy J. Lauver,et al.  Do psychosocial and study skill factors predict college outcomes? A meta-analysis. , 2004, Psychological bulletin.

[42]  Sandra Katz,et al.  Modeling the Student in Sherlock II , 1992 .

[43]  Megan E. Bird,et al.  Competition, cooperation, satisfaction, and the performance of complex tasks among air force cadets , 1990 .

[44]  Harold F. O'Neil,et al.  Perspectives on computer-based performance assessment of problem solving , 1999 .

[45]  Marshall G. Jones,et al.  Using Metacognitive Theories to Design User Interfaces for Computer-Based Learning. , 1995 .

[46]  Gregory K. W. K. Chung,et al.  Reliability and validity of a computer-based knowledge mapping system to measure content understanding , 1999 .

[47]  Ton de Jong,et al.  Scientific Discovery Learning with Computer Simulations of Conceptual Domains , 1998 .

[48]  J. Frederiksen,et al.  Inquiry, Modeling, and Metacognition: Making Science Accessible to All Students , 1998 .

[49]  R. Mayer,et al.  Engaging students in active learning: The case for personalized multimedia messages. , 2000 .

[50]  Thomas W. Malone,et al.  Toward a Theory of Intrinsically Motivating Instruction , 1981, Cogn. Sci..

[51]  Chien Chou,et al.  The effect of navigation map types and cognitive styles on learners' performance in a computer-networked hypertext learning system , 1998 .

[52]  D. Gopher,et al.  Transfer of Skill from a Computer Game Trainer to Flight , 1994 .

[53]  Lloyd P. Rieber,et al.  To Teach or Not to Teach? Comparing the Use of Computer-Based Simulations in Deductive versus Inductive Approaches to Learning with Adults in Science , 1995 .

[54]  Robert J. Mislevy,et al.  Psychometric Principles in Student Assessment , 2003 .

[55]  David M. Moore,et al.  The Effect of Navigation Tools on Learners' Achievement and Attitude in a Hypermedia Environment , 1999 .

[56]  Richard E. Mayer,et al.  Computer-Based Assessment of Problem Solving. , 1999 .

[57]  Jeffrey A. Jordan,et al.  Visual attention: Individual differences in training and predicting complex task performance , 1995 .

[58]  Harold F. O'Neil,et al.  A Technology to Support Leader Development: Computer Games. , 2004 .

[59]  R. Mayer,et al.  A Split-Attention Effect in Multimedia Learning: Evidence for Dual Processing Systems in Working Memory , 1998 .

[60]  C. Quinn.  Computers for cognitive research: A HyperCard adventure game , 1991 .

[61]  John A. Ellis,et al.  An Evaluation of Three Computer-Based Instructional Strategies in Basic Electricity and Electronics Training , 2000 .

[62]  Ton de Jong,et al.  KM QUEST: A collaborative Internet-based simulation game , 2003 .