A META-ANALYTIC EXAMINATION OF THE INSTRUCTIONAL EFFECTIVENESS OF COMPUTER-BASED SIMULATION GAMES

Interactive cognitive complexity theory suggests that simulation games are more effective than other instructional methods because they simultaneously engage trainees’ affective and cognitive processes (Tennyson & Jorczak, 2008). Meta-analytic techniques were used to examine the instructional effectiveness of computer-based simulation games relative to a comparison group (k= 65, N= 6,476). Consistent with theory, posttraining self-efficacy was 20% higher, declarative knowledge was 11% higher, procedural knowledge was 14% higher, and retention was 9% higher for trainees taught with simulation games, relative to a comparison group. However, the results provide strong evidence of publication bias in simulation games research. Characteristics of simulation games and the instructional context also moderated the effectiveness of simulation games. Trainees learned more, relative to a comparison group, when simulation games conveyed course material actively rather than passively, trainees could access the simulation game as many times as desired, and the simulation game was a supplement to other instructional methods rather than stand-alone instruction. However, trainees learned less from simulation games than comparison instructional methods when the instruction the comparison group received as a substitute for the simulation game actively engaged them in the learning experience.

[1]  J. Piaget The Psychology Of Intelligence , 1951 .

[2]  Rubin Gotesky,et al.  On knowing; essays for the left hand , 1963 .

[3]  On the Scientific Research for the Improvement of the Environment of Cities on the Deltaic Areas , 1966 .

[4]  B. Everitt,et al.  Statistical methods for rates and proportions , 1973 .

[5]  J. Fleiss Statistical methods for rates and proportions , 1974 .

[6]  Robert Sommer,et al.  Experience and Education. , 1974 .

[7]  Robert S. Ellinger,et al.  Computer-Assisted And Lecture Instruction: A Comparative Experiment , 1976 .

[8]  A. Bandura Self-efficacy: toward a unifying theory of behavioral change. , 1977, Psychological review.

[9]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[10]  G. Glass,et al.  Meta-analysis in social research , 1981 .

[11]  Thomas W. Malone,et al.  Toward a Theory of Intrinsically Motivating Instruction , 1981, Cogn. Sci..

[12]  David W. Stockburger Evaluation of Three Simulation Exercises in an Introductory Statistics Course. , 1982 .

[13]  R. Clark Reconsidering Research on Learning from Media , 1983 .

[14]  James E. Driskell,et al.  Microcomputer Videogame Based Training. , 1984 .

[15]  R. Noe Trainees' Attributes and Attitudes: Neglected Influences on Training Effectiveness , 1986 .

[16]  G. Phillip Cartwright,et al.  Effectiveness of Computer-Assisted Special Education Administration Simulations , 1986 .

[17]  Ann Majchrzak,et al.  Applied Social Research Methods Series. Vol. 2: The Integrative Research Review; A Systematic Approach. , 1986 .

[18]  Effects of Modes of Computer-Assisted Instruction on Conceptual Understanding and Achievement of College Students Exhibiting Individual Differences in Learning: A Pilot Study. , 1987 .

[19]  E. Deci,et al.  The support of autonomy and the control of behavior. , 1987, Journal of personality and social psychology.

[20]  Timothy T. Baldwin,et al.  TRANSFER OF TRAINING: A REVIEW AND DIRECTIONS FOR FUTURE RESEARCH , 1988 .

[21]  Jeffrey Gorrell,et al.  Effects of Computer-Simulated Behavior Analysis on Pre-Service Teachers' Problem Solving , 1989 .

[22]  John E. Laird,et al.  Symbolic architectures for cognition , 1989 .

[23]  Dee Anna Smith Willis Development and evaluation of IRIS: a computer simulation to teach preservice teachers to administer an informal reading inventory , 1989 .

[24]  H. Strang,et al.  Impact of microcomputer simulation training on student-clinicians' ability to effectively intervene with preschool stutterers , 1989 .

[25]  Valerie J. Shute,et al.  A Large-Scale Evaluation of an Intelligent Discovery World: Smithtown , 1990, Interact. Learn. Environ..

[26]  Ellen M. Whitener,et al.  Confusion of confidence intervals and credibility intervals in meta-analysis. , 1990 .

[27]  Mary W. Gray,et al.  The effect of simulation software on students' attitudes and understanding in introductory statistics , 1991 .

[28]  William H. Glick,et al.  Understanding Technology-Structure Relationships: Theory Development and Meta-Analytic Theory Testing , 1991 .

[29]  K. Lawson,et al.  Using a Computer-Simulation Program and a Traditional Approach to Teach Pharmacy Financial Management. , 1991 .

[30]  Chen-Lin C. Kulik,et al.  Effectiveness of computer-based instruction: An updated analysis. , 1991 .

[31]  George Brant,et al.  Which Comes First the Simulation or the Lecture? , 1991 .

[32]  P. Sleight,et al.  Publication bias , 1991, The Lancet.

[33]  A. Bandura Social cognitive theory of self-regulation☆ , 1991 .

[34]  Michael Frese,et al.  Errors in Training Computer Skills: on the Positive Function of Errors , 1991, Hum. Comput. Interact..

[35]  Anu A. Gokhale Effectiveness of Computer Simulation versus Lab and Sequencing of Instruction in Teaching Logic Circuits. , 1991 .

[36]  Computer-aided instruction in a physiological psychology course , 1992 .

[37]  M. Lepper,et al.  Effects of fantasy contexts on children's learning and motivation: making learning more fun. , 1992, Journal of personality and social psychology.

[38]  J. V. Faryniarz,et al.  Effectiveness of Microcomputer Simulations in Stimulating Environmental Problem Solving by Community College Students. , 1992 .

[39]  C. Douglas Wetzel,et al.  The Effectiveness of Games for Educational Purposes: A Review of Recent Research , 1992 .

[40]  Mark W. Lipsey,et al.  The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. , 1993, The American psychologist.

[41]  Saeid Moslehpour,et al.  A comparison of achievement resulting from learning electronics concepts by computer simulation versus traditional laboratory instruction , 1993 .

[42]  Rex A. Thomas,et al.  An Investigation of Manipulative Models on the Learning of Programming Loops , 1993 .

[43]  E. Salas,et al.  Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. , 1993 .

[44]  A. Bandura Perceived Self-Efficacy in Cognitive Development and Functioning , 1993, Educational Psychologist.

[45]  D. Gopher,et al.  Transfer of Skill from a Computer Game Trainer to Flight , 1994 .

[46]  Gustavo A. Ortiz EFFECTIVENESS OF PC-BASED FLIGHT SIMULATION / , 1994 .

[47]  Harold F. O'Neil,et al.  Technology assessment in education and training , 1994 .

[48]  R. Clark Media will never influence learning , 1994 .

[49]  Ann E. Barron,et al.  Teaching Telecommunications: A Comparison between Video and Computer-Based Instruction. , 1994 .

[50]  John Waddick Case Study: The Use of a Hypercard Simulation to Aid in the Teaching of Laboratory Apparatus Operation , 1994 .

[51]  Harold F. O'Neil,et al.  Meta-analytic studies of findings on computer-based instruction. , 1994 .

[52]  E. Salas,et al.  Military Team Research: 10 Years of Progress , 1995 .

[53]  Marios C. Angelides,et al.  Evaluating intelligent tutoring with gaming-simulations , 1995, WSC '95.

[54]  Raenel E. Kinkade Evaluation of a Computer Simulation in a Therapeutics Case Discussion. , 1995 .

[55]  Psychological foundations for instructional design theory: Cognitive complexity theory. , 1995 .

[56]  Winfred Arthur,et al.  Development of a new outlier statistic for meta-analytic data , 1995 .

[57]  M. Lepper,et al.  Intrinsic motivation and the process of learning: Beneficial effects of contextualization, personalization, and choice. , 1996 .

[58]  Katrina E. Ricci,et al.  Do Computer-Based Games Facilitate Knowledge Acquisition and Retention? , 1996 .

[59]  A. Bandura Self-Efficacy: The Exercise of Control , 1997, Journal of Cognitive Psychotherapy.

[60]  E. Salas,et al.  Methods, tools, and strategies for team training. , 1997 .

[61]  J. Webster,et al.  Teaching Effectiveness in Technology-Mediated Distance Learning , 1997 .

[62]  Ying-Shao Hsu,et al.  Development and formative evaluation of an instructional simulation of adiabatic processes , 1997 .

[63]  Randall Wayne Bower An investigation of a manipulative simulation in the learning of recursive programming , 1998 .

[64]  Sandra L. Fisher,et al.  DIFFERENTIAL EFFECTS OF LEARNER EFFORT AND GOAL ORIENTATION ON TWO LEARNING OUTCOMES , 1998 .

[65]  Chiu-Hsiang Chou,et al.  The effectiveness of using multimedia computer simulations coupled with social constructivist pedagogy in a college introductory physics classroom , 1998 .

[66]  E Salas,et al.  It is not how much you have but how you use it: toward a rational use of simulation to support aviation training. , 1998, The International journal of aviation psychology.

[67]  Y. Liao Effects of hypermedia on students' achievement: a meta-analysis , 1999 .

[68]  John J. Hirschbuhl,et al.  Does interactive multimedia promote achievement and higher level thinking skills for today's science students? , 1999, Br. J. Educ. Technol..

[69]  June Lee,et al.  Effectiveness of Computer-Based Instructional Simulation: A Meta Analysis. , 1999 .

[70]  G. Stamou,et al.  An Investigation of the Effectiveness of Computer Simulation Programs as Tutorial Tools for Teaching Population Ecology at University. , 1999 .

[71]  John Sweller,et al.  Instructional Design in Technical Areas , 1999 .

[72]  H. Schwid,et al.  Use of a computerized advanced cardiac life support simulator improves retention of advanced cardiac life support guidelines better than a textbook review. , 1999, Critical care medicine.

[73]  Viswanath Venkatesh,et al.  Creating an effective training environment for enhancing telework , 2000, Int. J. Hum. Comput. Stud..

[74]  John A. Ellis,et al.  An Evaluation of Three Computer-Based Instructional Strategies in Basic Electricity and Electronics Training , 2000 .

[75]  Parshotam Dass,et al.  Improving Students’ Self-Efficacy in Strategic Management: The Relative Impact of Cases and Simulations , 2000 .

[76]  Marcie N. Desrochers,et al.  An Evaluation of Simulations in Developmental Disabilities (SIDD): Instructional Software that Provides Practice in Behavioral Assessment and Treatment Decisions , 2001 .

[77]  E. Salas,et al.  The science of training: a decade of progress. , 2001, Annual review of psychology.

[78]  Robert A. Reiser,et al.  Trends and issues in instructional design and technology , 2002 .

[79]  Ian E. Hughes,et al.  Alternatives to Laboratory Practicals – Do They Meet the Needs? , 2001 .

[80]  William H. Ross,et al.  Interactive Video Negotiator Training: A Preliminary Evaluation of the McGill Negotiation Simulator , 2001 .

[81]  Theresa Pesl Murphrey,et al.  Evaluation of a Computer-Based, Asynchronous Activity on Student Learning of Leadership Concepts. , 2002 .

[82]  S. Kozlowski,et al.  Adaptive Guidance: Enhancing Self-Regulation, Knowledge, and Performance in Technology-Based Training , 2002 .

[83]  R. Sternberg,et al.  The Psychology of Intelligence , 2002 .

[84]  Jong Hoon Kim,et al.  Learning by Computer Simulation Does Not Lead to Better Test Performance on Advanced Cardiac Life Support Than Textbook Study. , 2002, The journal of education in perioperative medicine : JEPM.

[85]  Won Oak Kim,et al.  Learning by computer simulation does not lead to better test performance than textbook study in the diagnosis and treatment of dysrhythmias. , 2002, Journal of clinical anesthesia.

[86]  James E. Driskell,et al.  Games, Motivation, and Learning: A Research and Practice Model , 2002 .

[87]  M. E. Gredler,et al.  EDUCATIONAL GAMES AND SIMULATIONS : A TECHNOLOGY IN SEARCH OF A ( RESEARCH ) PARADIGM , 2002 .

[88]  Piers Steel,et al.  Comparing meta-analytic moderator estimation techniques under realistic conditions. , 2002, The Journal of applied psychology.

[89]  Steve Pettifer,et al.  Transfer of route learning from virtual to real environments. , 2003, Journal of experimental psychology. Applied.

[90]  Mark W. Lipsey,et al.  Those Confounded Moderators in Meta-Analysis: Good, Bad, and Ugly , 2003 .

[91]  Sheila Danko,et al.  Disentangling Achievement Orientation and Goal Setting: Effects on Self-Regulatory Processes , 2003 .

[92]  Gary J. Summers Today’s Business Simulation Industry , 2004 .

[93]  R. Mayer Should there be a three-strikes rule against pure discovery learning? The case for guided methods of instruction. , 2004, The American psychologist.

[94]  J. Sweller Instructional Design Consequences of an Analogy between Evolution by Natural Selection and Human Cognitive Architecture , 2004 .

[95]  Max M. North,et al.  Immersive visualization tool for pedagogical practices of computer science concepts: a pilot study , 2004 .

[96]  Rex C. Mitchell Combining Cases and Computer Simulations in Strategic Management Courses , 2004 .

[97]  Eduardo Salas,et al.  Can PC-Based Systems Enhance Teamwork in the Cockpit? , 2005 .

[98]  M. Frese,et al.  Self-regulation in error management training: emotion control and metacognition as mediators of performance effects. , 2005, The Journal of applied psychology.

[99]  Francis M. Dwyer,et al.  The Effect of Online Gaming, Cognition and Feedback Type in Facilitating Delayed Achievement of Different Learning Objectives , 2005 .

[100]  M. D. Dunnette Handbook of Industrial and Organizational Psychology , 2005 .

[101]  H. O'Neil,et al.  Classification of learning outcomes: evidence from the computer games literature , 2005 .

[102]  Robert T Hays,et al.  The Effectiveness of Instructional Games: A Literature Review and Discussion , 2005 .

[103]  Hsin-Hui Chen,et al.  A formative evaluation of the training effectiveness of a computer game , 2005 .

[104]  John V. Dempsey,et al.  The effects of web-based animated demonstration and guided simulation on time and performance in business software applications training , 2005 .

[105]  Yong Zhao,et al.  What Makes the Difference? A Practical Analysis of Research on the Effectiveness of Distance Education , 2005, Teachers College Record: The Voice of Scholarship in Education.

[106]  S. Kozlowski,et al.  Disentangling achievement orientation and goal setting: effects on self-regulatory processes. , 2006, The Journal of applied psychology.

[107]  Robert A. Wisher,et al.  THE COMPARATIVE EFFECTIVENESS OF WEB-BASED AND CLASSROOM INSTRUCTION: , 2006 .

[108]  Josh Schollmeyer Games Get Serious , 2006 .

[109]  Michelene T. H. Chi,et al.  Simulation versus Text: Acquisition of Implicit and Explicit Information , 2006 .

[110]  Michelle F. Wright,et al.  Computer Gaming and Interactive Simulations for Learning: A Meta-Analysis , 2006 .

[111]  Erdat Cataloglu Open Source Software in Teaching Physics: A Case Study on Vector Algebra and Visual Representations. , 2006 .

[112]  Alvin C. M. Kwan,et al.  Online vocabulary games as a tool for teaching and learning English vocabulary , 2006 .

[113]  Tammy L. Rapp,et al.  Evaluating an Individually Self-Administered Generic Teamwork Skills Training Program Across Time and Levels , 2007 .

[114]  Kurt Kraiger,et al.  A meta-analytic investigation of learner control: Old findings and new directions. , 2007 .

[115]  Ricki S. Loar The Impact of a Computer Simulated Case Study on Nurse Practitioner Students' Declarative Knowledge and Clinical Performance , 2007 .

[116]  L. Bizo,et al.  DTkid: Interactive Simulation Software for Training Tutors of Children with Autism , 2007, Journal of autism and developmental disorders.

[117]  Martin Ebner,et al.  Successful implementation of user-centered game based learning in higher education: An example from civil engineering , 2007, Comput. Educ..

[118]  Shawn M. Fitzgerald,et al.  The Effectiveness of Technology Use in Statistics Instruction in Higher Education: A Meta-Analysis , 2008 .

[119]  Mark D. Griffiths,et al.  Experiences of Time Loss among Videogame Players: An Empirical Study , 2007, Cyberpsychology Behav. Soc. Netw..

[120]  Z. C. Zacharia,et al.  Comparing and combining real and virtual experimentation: an effort to enhance students' conceptual understanding of electric circuits , 2007, J. Comput. Assist. Learn..

[121]  Ray S. Perez,et al.  Computer Games and Team and Individual Learning , 2007 .

[122]  J. D. Fletcher,et al.  What Research Has to Say about Designing Computer Games for Learning. , 2007 .

[123]  Katherine Ely,et al.  Do Second-Generation Models Have More to Offer? , 2008, Industrial and Organizational Psychology.

[124]  Celal Bayrak Effects of Computer Simulations Programs On University Students’ Achievments In Physics , 2008 .

[125]  Robert D. Tennyson,et al.  A conceptual framework for the empirical study of games , 2008 .

[126]  Leslie Jarmon,et al.  Understanding project‐based learning in Second Life with a pedagogy, training, and assessment trio , 2008 .

[127]  S. Kozlowski,et al.  Active learning: effects of core training design elements on self-regulatory processes, learning, and adaptability. , 2008, The Journal of applied psychology.

[128]  Andrew Moshirnia The influence of information delivery systems in modified video games on learning , 2008 .

[129]  Katherine Ely,et al.  A review and meta-analysis of the nomological network of trainee reactions. , 2008, The Journal of applied psychology.

[130]  M. Frese,et al.  Effectiveness of error management training: a meta-analysis. , 2008, The Journal of applied psychology.

[131]  B. Fritzsche,et al.  Game on: the impact of game features in computer-based training , 2008 .

[132]  Daniel B. Horn,et al.  The roles of task difficulty and prior videogame experience on performance and motivation in instructional videogames , 2008, Comput. Hum. Behav..

[133]  Xiao-Hua Zhou,et al.  Statistical Methods for Meta‐Analysis , 2008 .

[134]  Steve W. J. Kozlowski,et al.  Current issues and future directions in simulation-based training in North America , 2008 .

[135]  Andreas Holzinger,et al.  Learning performance with interactive simulations in medical education: Lessons learned from results of learning complex physiological models with the HAEMOdynamics SIMulator , 2009, Comput. Educ..

[136]  Gary C. Brown,et al.  Comparative effectiveness , 2009, Current opinion in ophthalmology.

[137]  Kara L. Orvis,et al.  Relationships Between Game Attributes and Learning Outcomes , 2009 .

[138]  Kristina N. Bauer,et al.  Self-Assessment of Knowledge: A Cognitive Learning or Affective Measure? , 2010 .

[139]  Robert A. Wisher,et al.  Designing Web-Based Training Courses to Maximize Learning , 2010 .

[140]  Traci Sitzmann,et al.  Sometimes you need a reminder: the effects of prompting self-regulation on regulatory processes, learning, and attrition. , 2010, The Journal of applied psychology.

[141]  Traci Sitzmann,et al.  Training and employee development for improved performance. , 2011 .