Evaluating Large-Scale Training Simulations

This article develops and describes a proposed training evaluation framework for large-scale training simulations (LSTS). These simulations may link together hundreds of participants at many different geographic locations and are more complex and costly than traditional training devices and simulators by an order of magnitude or more. The evaluation framework is based on a review and analysis of the literature pertaining to military training evaluation and ideas that have percolated up in the past dozen years from a small number of scientists engaged in LSTS evaluation. The analysis focuses on key evaluation questions: What is the purpose of the evaluation? Whose training is being evaluated? What is being evaluated? Where is the evaluation conducted? What evaluation methods are used? What are the evaluation criteria? What is the timing of evaluation events? The framework developed ties together evaluation events with prescribed objectives, timing, methods, and relevant evaluation criteria (dependent variables). It is intended to apply to any large-scale virtual, constructive, or advanced distributed simulation.

[1]  Robert L. Craig,et al.  Training and development handbook: A guide to human resource development , 1976 .

[2]  G. Alliger,et al.  KIRKPATRICK'S LEVELS OF TRAINING CRITERIA: THIRTY YEARS LATER , 1989 .

[3]  Henry Simpson.  Evaluating Large-Scale Training Simulations. Volume 1. Reference Manual , 1999 .

[4]  Bruce C. Leibrecht.  An Integrated Database of Unit Training Performance: Description an Lessons Learned. , 1997 .

[5]  Daniel J. Dwyer,et al.  Appendices. Recommendations for Planning and Conducting Multi-Service TacticaL Training With Distributed Interactive Simulation Technology , 1997 .

[6]  David W. Bessemer,et al.  Training Research with Distributed Interactive Simulations: Lessons Learned from Simulation Networking , 1994 .

[7]  Jon J. Fallesen,et al.  Review of Command Group Training Measurement Methods , 1988 .

[8]  L. Sjodahl.  Evaluation of Training. , 1975 .

[9]  E. Salas,et al.  Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. , 1993 .

[10]  James D. Baker,et al.  Field Testing: The Delicate Compromise1 , 1974 .

[11]  Martha Reeves,et al.  Evaluation of Training , 1993 .

[12]  Winston Bennett,et al.  A META-ANALYSIS OF THE RELATIONS AMONG TRAINING CRITERIA , 1997 .

[13]  D. Campbell.  Factors relevant to the validity of experiments in social settings. , 1957, Psychological bulletin.

[14]  Michelle Bailey,et al.  Utility of Modeling and Simulation in the Department of Defense: Initial Data Collection. , 1996 .

[15]  E. A. Alluisi,et al.  The Development of Technology for Collective Training: SIMNET, a Case History , 1991 .

[16]  Henry L. Taylor,et al.  The Value of Simulation for Training. , 1994 .

[17]  Daniel B. Levine,et al.  The Cost and Effectiveness of the Multi-Service Distributed Training Testbed (MDT2) for Training Close Air Support. , 1997 .