A meta-analysis of virtual reality training programs

Virtual reality (VR) is the three-dimensional digital representation of a real or imagined space with interactive capabilities. The application of VR for organizational training purposes has been surrounded by much fanfare; however, mixed results have been provided for the effectiveness of VR training programs, and the attributes of effective VR training programs are still unknown. To address these issues, we perform a meta-analysis of controlled experimental studies that tests the effectiveness of VR training programs. We obtain an estimate of the overall effectiveness of VR training programs, and we identify features of VR training programs that systematically produce improved results. Our meta-analytic findings support that VR training programs produce better outcomes than tested alternatives. The results also show that few moderating effects were significant. The applied display hardware, input hardware, and inclusion of game attributes had non-significant moderating effects; however, task-technology fit and aspects of the research design did influence results. We suggest that task-technology fit theory is an essential paradigm for understanding VR training programs, and no set of VR technologies is “best” across all contexts. Future research should continue studying all types of VR training programs, and authors should more strongly integrate research and theory on employee training and

[1]  J. Sánchez-Meca,et al.  TESTING CONTINUOUS MODERATORS IN META-ANALYSIS : A COMPARISON OF PROCEDURES , 1998 .

[2]  Stefan Greuter,et al.  Towards personalised, gamified systems: an investigation into game design, personality and player typologies , 2013, IE.

[3]  R. Orwin.  A fail-safe N for effect size in meta-analysis. , 1983 .

[4]  Steve Pettifer,et al.  Absorption, dissociation, locus of control and presence in virtual reality , 2007, Comput. Hum. Behav..

[5]  Mike W.-L. Cheung,et al.  Meta-Analysis: A Structural Equation Modeling Approach , 2015 .

[6]  Michela Ott,et al.  A LITERATURE REVIEW ON IMMERSIVE VIRTUAL REALITY IN EDUCATION: STATE OF THE ART AND PERSPECTIVES. , 2015, 11th International Conference eLearning and Software for Education.

[7]  D. Kolb,et al.  Experiential Learning Theory: Previous Research and New Directions , 2000 .

[8]  James G. Field,et al.  Correlational effect size benchmarks. , 2015, The Journal of applied psychology.

[9]  H. Kenngott,et al.  LapTrain: multi-modality training curriculum for laparoscopic cholecystectomy—results of a randomized controlled trial , 2018, Surgical Endoscopy.

[10]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .

[11]  Jiejie Zhu,et al.  Virtual Reality and Mixed Reality for Virtual Learning Environments , 2006, Edutainment.

[12]  L. Hedges,et al.  Introduction to Meta‐Analysis , 2009, International Coaching Psychology Review.

[13]  Timothy T. Baldwin,et al.  Transfer of Training: A Meta-Analytic Review , 2010 .

[14]  Mohamed Jamal Zemerly,et al.  Learner-Centric Context-Aware Mobile Learning , 2007 .

[15]  Christopher L. Groves,et al.  Video Games Do Indeed Influence Children and Adolescents’ Aggression, Prosocial Behavior, and Academic Performance , 2015, Perspectives on psychological science : a journal of the Association for Psychological Science.

[16]  Johann Füller,et al.  Avatar-Based Innovation: Consequences of the Virtual Co-Creation Experience , 2010, 2010 43rd Hawaii International Conference on System Sciences.

[17]  Steve W. J. Kozlowski,et al.  A Psychological Fidelity Approach to Simulation-Based Training: Theory, Research, and Principles , 2004 .

[18]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[19]  Zahira Merchant,et al.  Exploring 3-D virtual reality technology for spatial ability and chemistry achievement , 2013, J. Comput. Assist. Learn..

[20]  Chun-Yen Chang,et al.  A cost-effective interactive 3D virtual reality system applied to military live firing training , 2016, Virtual Reality.

[21]  Ruimin Shen,et al.  A Mobile Live Video Learning System for Large-Scale Learning—System Design and Evaluation , 2010, IEEE Transactions on Learning Technologies.

[22]  Teodor P Grantcharov,et al.  The impact of constructive feedback on training in gastrointestinal endoscopy using high-fidelity virtual-reality simulation: a randomised controlled trial , 2009, Gut.

[23]  Patrice L. Weiss,et al.  Effectiveness of virtual reality for teaching street-crossing skills to children and adolescents with autism , 2008 .

[24]  Sharona T. Levy,et al.  Now I know how! The learning process of medication administration among nursing students with non-immersive desktop virtual reality simulation , 2017, Comput. Educ..

[25]  John E. Hunter,et al.  Fixed Effects vs. Random Effects Meta‐Analysis Models: Implications for Cumulative Research Knowledge , 2000 .

[26]  M. Borenstein,et al.  Publication Bias in Meta-Analysis , 2006 .

[27]  John Short,et al.  The social psychology of telecommunications , 1976 .

[28]  Moon J. Lee,et al.  Do shy people feel less communication apprehension online? The effects of virtual reality on the relationship between personality characteristics and communication outcomes , 2011, Comput. Hum. Behav..

[29]  Thomas S. Redick,et al.  Working Memory Training May Increase Working Memory Capacity but Not Fluid Intelligence , 2013, Psychological science.

[30]  David C Schwebel,et al.  Using virtual reality to train children in safe street-crossing skills , 2010, Injury Prevention.

[31]  D. Kirkpatrick.  Techniques for evaluating training programs , 1979 .

[32]  Traci Sitzmann.  A META-ANALYTIC EXAMINATION OF THE INSTRUCTIONAL EFFECTIVENESS OF COMPUTER-BASED SIMULATION GAMES , 2011 .

[33]  Christopher L. Camp,et al.  Improving Resident Performance in Knee Arthroscopy: A Prospective Value Assessment of Simulators and Cadaveric Skills Laboratories. , 2016, The Journal of bone and joint surgery. American volume.

[34]  J. Ioannidis,et al.  The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration , 2009, Annals of Internal Medicine [serial online].

[35]  Susanne M. Jaeggi,et al.  Adaptive n-back training does not improve fluid intelligence at the construct level: Gains on individual tests suggest that training may enhance visuospatial processing , 2013 .

[36]  Eduardo Salas,et al.  Performance Measurement in Simulation-Based Training , 2009 .

[37]  Wolfgang Viechtbauer,et al.  Outlier and influence diagnostics for meta‐analysis , 2010, Research synthesis methods.

[38]  John E. Hunter,et al.  Tacit Knowledge, Practical Intelligence, General Mental Ability, and Job Knowledge , 1993 .

[39]  Eduardo Salas,et al.  The Science of Training and Development in Organizations , 2012, Psychological science in the public interest : a journal of the American Psychological Society.

[40]  Joseph Cohn,et al.  An examination of virtual environment training fidelity on training effectiveness , 2017, Int. J. Learn. Technol..

[41]  Bruce H. Thomas,et al.  Immersive Collaborative Analysis of Network Connectivity: CAVE-style or Head-Mounted Display? , 2017, IEEE Transactions on Visualization and Computer Graphics.

[42]  Matt C. Howard.  Virtual Reality Interventions for Personal Development: A Meta-Analysis of Hardware and Software , 2019, Hum. Comput. Interact..

[43]  Thenkurussi Kesavadas,et al.  Can image-based virtual reality help teach anatomy? , 2010, Journal of endourology.

[44]  D. Moher,et al.  Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement , 2009, BMJ : British Medical Journal.

[45]  Theo Stijnen,et al.  Advanced methods in meta‐analysis: multivariate approach and meta‐regression , 2002, Statistics in medicine.

[46]  Shwu-Huey Wang,et al.  Applying a 3D virtual learning environment to facilitate student's application ability - The case of marketing , 2011, Comput. Hum. Behav..

[47]  Elske Ammenwerth,et al.  BMC Medical Informatics and Decision Making , 2022 .

[48]  P. Shekelle,et al.  Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement , 2015, Systematic Reviews.

[49]  R. Aggarwal,et al.  Virtual reality training for surgical trainees in laparoscopic surgery. , 2009, The Cochrane database of systematic reviews.

[50]  Herman Aguinis,et al.  Revisiting the file drawer problem in meta‐analysis: An assessment of published and nonpublished correlation matrices. , 2012 .

[51]  Sven Kepes,et al.  Meta-analytic Reviews in the Organizational Sciences: Two Meta-analytic Schools on the Way to MARS (the Meta-analytic Reporting Standards) , 2013 .

[52]  Peter Åsman,et al.  Virtual reality cataract surgery training: learning curves and concurrent validity , 2012, Acta ophthalmologica.

[53]  Nirit Gavish,et al.  Physical Fidelity Versus Cognitive Fidelity Training in Procedural Skills Acquisition , 2011, Hum. Factors.

[54]  Teodor P Grantcharov,et al.  Learning curves and impact of previous operative experience on performance on a virtual reality simulator to test laparoscopic surgical skills. , 2003, American journal of surgery.

[55]  Fady T. Charbel,et al.  Virtual reality training in neurosurgery: Review of current status and future applications , 2011, Surgical neurology international.

[56]  Joan McComas,et al.  Effectiveness of Virtual Reality for Teaching Pedestrian Safety , 2002, Cyberpsychology Behav. Soc. Netw..

[57]  Michael A. Gigante,et al.  Virtual Reality: Definitions, History and Applications , 1993, Virtual Reality Systems.

[58]  Franco Tecchia,et al.  Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks , 2015, Interact. Learn. Environ..

[59]  Matt C. Howard,et al.  A meta-analysis of virtual reality training programs for social skill development , 2020, Comput. Educ..

[60]  R. Satava,et al.  Virtual Reality Training Improves Operating Room Performance: Results of a Randomized, Double-Blinded Study , 2002, Annals of surgery.

[61]  M. Moshell,et al.  Three views of virtual reality: virtual environments in the US military , 1993, Computer.

[62]  Timothy T. Baldwin,et al.  TRANSFER OF TRAINING: A REVIEW AND DIRECTIONS FOR FUTURE RESEARCH , 1988 .

[63]  T. Arulampalam,et al.  Virtual reality training in laparoscopic surgery: A systematic review & meta-analysis. , 2016, International journal of surgery.

[64]  Martin Skitmore,et al.  Multiuser Virtual Safety Training System for Tower Crane Dismantlement , 2012, J. Comput. Civ. Eng..

[65]  Tor Söderström,et al.  Collaborative learning with screen-based simulation in health care education: an empirical study of collaborative patterns and proficiency development , 2011, J. Comput. Assist. Learn..

[66]  Matt C. Howard.  A meta-analysis and systematic literature review of virtual reality rehabilitation programs , 2017, Comput. Hum. Behav..

[67]  Eduardo Salas,et al.  Toward a Taxonomy Linking Game Attributes to Learning , 2012 .

[68]  M. Frese,et al.  Effectiveness of error management training: a meta-analysis. , 2008, The Journal of applied psychology.

[69]  Simon J. Julier,et al.  The software architecture of a real-time battlefield visualization virtual environment , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[70]  Yin-Leng Theng,et al.  The Use of Videogames, Gamification, and Virtual Environments in the Self-Management of Diabetes: A Systematic Review of Evidence. , 2015, Games for health journal.

[71]  Carlos Delgado Kloos,et al.  Impact of an augmented reality system on students' motivation for a visual art course , 2013, Comput. Educ..

[72]  D. Kirkpatrick.  Evaluating Training Programs , 1994 .

[73]  Nancy J Hogle,et al.  Documenting a learning curve and test-retest reliability of two tasks on a virtual reality training simulator in laparoscopic surgery. , 2007, Journal of surgical education.

[74]  Steve Hand,et al.  The Effect of Mobile Phone Screen Size on Video Based Learning , 2008, J. Softw..

[75]  B. Bushman,et al.  Methodological and Reporting Errors in Meta-Analytic Reviews Make Other Meta-Analysts Angry , 2015, Perspectives on psychological science : a journal of the Association for Psychological Science.

[76]  Frederic D. McKenzie,et al.  Design and Comparison of Immersive Interactive Learning and Instructional Techniques for 3D Virtual Laboratories , 2015, PRESENCE: Teleoperators and Virtual Environments.

[77]  Michael Zyda,et al.  From visual simulation to virtual reality to games , 2005, Computer.

[78]  Isabel C. Green,et al.  A Pilot Study of Surgical Training Using a Virtual Robotic Surgery Simulator , 2013, JSLS : Journal of the Society of Laparoendoscopic Surgeons.

[79]  Thomas G. Plante,et al.  Might virtual reality promote the mood benefits of exercise? , 2003, Comput. Hum. Behav..

[80]  Marcus A. Butavicius,et al.  Evaluation of a Virtual Reality Parachute Training Simulator: Assessing Learning in an Off-Course Augmented Feedback Training Schedule , 2012 .

[81]  Nikolaos Pellas,et al.  The influence of computer self-efficacy, metacognitive self-regulation and self-esteem on student engagement in online learning programs: Evidence from the virtual world of Second Life , 2014, Comput. Hum. Behav..

[82]  F. Schmidt.  Statistical and measurement pitfalls in the use of meta-regression in meta-analysis , 2017 .

[83]  Erlend Fagertun Hofstad,et al.  Lack of transfer of skills after virtual reality simulator training with haptic feedback , 2017, Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy.

[84]  D. Man.  Virtual reality-based cognitive training for drug abusers: A randomised controlled trial , 2020, Neuropsychological rehabilitation.

[85]  Suzanne Jak,et al.  Meta-Analytic Structural Equation Modelling , 2015 .

[86]  Dimitrios Buhalis,et al.  Serious games and the gamification of tourism , 2017 .

[87]  Franck Ganier,et al.  Evaluation of procedural learning transfer from a virtual environment to a real situation: a case study on tank maintenance training , 2014, Ergonomics.

[88]  Neil Vaughan,et al.  A review of virtual reality based training simulators for orthopaedic surgery. , 2016, Medical engineering & physics.

[89]  N. J. Cohen,et al.  Different Memory Systems Underlying Acquisition of Procedural and Declarative Knowledge a , 1985, Annals of the New York Academy of Sciences.

[90]  S. Thompson,et al.  How should meta‐regression analyses be undertaken and interpreted? , 2002, Statistics in medicine.

[91]  Carolina Cruz-Neira,et al.  Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE , 2023 .

[92]  Mike W-L Cheung,et al.  Random‐effects models for meta‐analytic structural equation modeling: review, issues, and illustrations , 2016, Research synthesis methods.

[93]  O. Bock,et al.  Can Driving-Simulator Training Enhance Visual Attention, Cognition, and Physical Functioning in Older Adults? , 2018, Journal of aging research.

[94]  Seung Hwan Lee,et al.  Assessing Google Cardboard virtual reality as a content delivery system in business classrooms , 2017 .

[95]  D. Kolb,et al.  Experiential Learning Theory: A Dynamic, Holistic Approach to Management Learning, Education and Development , 2009 .

[96]  M. Rosenberg,et al.  THE FILE‐DRAWER PROBLEM REVISITED: A GENERAL WEIGHTED METHOD FOR CALCULATING FAIL‐SAFE NUMBERS IN META‐ANALYSIS , 2005, Evolution; international journal of organic evolution.

[97]  Gilles E. Gignac,et al.  Effect size guidelines for individual differences researchers , 2016 .

[98]  Charlotte Ringsted,et al.  Simulation-based training for thoracoscopic lobectomy: a randomized controlled trial , 2014, Surgical Endoscopy.

[99]  Elizabeth Tipton,et al.  A history of meta‐regression: Technical, conceptual, and practical developments between 1974 and 2018 , 2019, Research synthesis methods.

[100]  David Wiley,et al.  Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy , 2000 .

[101]  Harold F. O'Neil,et al.  Meta-analytic studies of findings on computer-based instruction. , 1994 .

[102]  Eduardo Salas,et al.  Making decisions under stress: Implications for individual and team training. , 1998 .

[103]  Richard N. Landers,et al.  Gamification Science, Its History and Future: Definitions and a Research Agenda , 2018 .

[104]  Hakan Tüzün,et al.  The effects of 3D multi-user virtual environments on freshmen university students' conceptual and spatial learning and presence in departmental orientation , 2016, Comput. Educ..

[105]  Deborah I. Fels,et al.  Gamification in theory and action: A survey , 2015, Int. J. Hum. Comput. Stud..

[106]  E. Salas,et al.  The science of training: a decade of progress. , 2001, Annual review of psychology.

[107]  R. Landers.  Developing a Theory of Gamified Learning , 2014 .

[108]  Richard M. Satava,et al.  An integrated medical virtual reality program. The military application , 1996 .

[109]  Daniel C. Krawczyk,et al.  Virtual Reality Social Cognition Training for children with high functioning autism , 2016, Comput. Hum. Behav..

[110]  Chung-Ho Su,et al.  3D Game-Based Learning System for Improving Learning Achievement in Software Engineering Curriculum , 2013 .

[111]  Jessica L. Wildman,et al.  Using Simulation-Based Training to Enhance Management Education , 2009 .

[112]  Pebertli Nils Alho Barata,et al.  Consolidating Learning in Power Systems: Virtual Reality Applied to the Study of the Operation of Electric Power Transformers , 2015, IEEE Transactions on Education.

[113]  Ulrike Cress,et al.  Virtual training: Making reality work? , 2022 .

[114]  A. Cuschieri,et al.  A Systematic Review of Virtual Reality Simulators for Robot-assisted Surgery. , 2016, European urology.

[115]  L. Hedges,et al.  Fixed- and random-effects models in meta-analysis. , 1998 .

[116]  James E. Driskell,et al.  Games, Motivation, and Learning: A Research and Practice Model , 2002 .

[117]  Benson Rosen,et al.  Training for virtual teams: An investigation of current practices and future needs , 2006 .

[118]  Dong-Hee Shin,et al.  Empathy and embodied experience in virtual environment: To what extent can virtual reality stimulate empathy and embodied experience? , 2018, Comput. Hum. Behav..

[119]  Nader N Massarweh,et al.  Laparoscopic virtual reality training: are 30 repetitions enough? , 2004, The Journal of surgical research.

[120]  Jonathan Steuer,et al.  Defining virtual reality: dimensions determining telepresence , 1992 .

[121]  Dale Goodhue,et al.  Task-Technology Fit and Individual Performance , 1995, MIS Q..

[122]  Doug A. Bowman,et al.  Evaluating Display Fidelity and Interaction Fidelity in a Virtual Reality Game , 2012, IEEE Transactions on Visualization and Computer Graphics.

[123]  M. Beier,et al.  THE EFFECTIVENESS OF ERROR MANAGEMENT TRAINING WITH WORKING‐AGED ADULTS , 2010 .

[124]  Wim Van den Noortgate,et al.  Estimation of the predictive power of the model in mixed-effects meta-regression: A simulation study. , 2014, The British journal of mathematical and statistical psychology.

[125]  Tovi Grossman,et al.  GamiCAD: a gamified tutorial system for first time autocad users , 2012, UIST.

[126]  Matt C. Howard,et al.  Refining and extending task-technology fit theory: Creation of two task-technology fit scales and empirical clarification of the construct , 2019, Inf. Manag..

[127]  Michael J. Taber.  Simulation fidelity and contextual interference in helicopter underwater egress training: An analysis of training and retention of egress skills , 2014 .

[128]  Kara L. Orvis,et al.  Relationships Between Game Attributes and Learning Outcomes , 2009 .