Task difficulty of virtual reality-based assessment tools compared to classical paper-and-pencil or computerized measures: A meta-analytic approach

Virtual reality-based assessment tools arise as a promising alternative for classic neuropsychological assessment with an increased level of ecological validity. Because virtual reality cognitive measures recreate tasks that resemble with the demands from the real world it is assumed that they require additional cognitive resources and are more difficult than classical paper-and-pencil or computerized measures. Although research has focused on comparing the performance obtained on virtual reality-based measures with classical paper-and-pencil or computerized measures, no meta-analysis has been conducted on this topic. Thirteen studies met our inclusion criteria: assessed any cognitive process using virtual reality and analogous classical or computerized assessment tools of the same process. Based on a random effects model, the results indicated a moderate effect size in favor of classical and computerized tests (g?=?-0.77) revealing an increased task difficulty in virtual reality. Overall, results from the current meta-analysis point out that cognitive performance obtained in virtual reality is poorer than the one in classical or computerized assessment which might suggest that tasks embedded in virtual reality have an increased level of complexity and difficulty and require additional cognitive resources. Meta-analysis revealed increased task difficulty for virtual reality measures.Subgroup analysis showed somehow different patterns for the three outcomes.Age, clinical status, and task performance indicator were significant moderators.

[1]  T. Parsons.  Virtual Simulations and the Second Life Metaverse: Paradigm Shift in Neuropsychological Assessment , 2011 .

[2]  Albert Rizzo,et al.  Comparison of Two VR Platforms for Virtual Reality Rehabilitation: Video Capture versus HMD. , 2005 .

[3]  R. Dean,et al.  TEST REVIEW: Dean C. Delis, Edith Kaplan & Joel H. Kramer, Delis Kaplan Executive Function System (D-KEFS), The Psychological Corporation, San Antonio, TX, 2001. $415.00 (complete kit) , 2006 .

[4]  A. Puente,et al.  Assessment of neuropsychological functioning. , 2013 .

[5]  G. Riva,et al.  Virtual reality as assessment tool in psychology. , 1997, Studies in health technology and informatics.

[6]  E. Kaplan,et al.  Delis-Kaplan Executive Function System , 2012 .

[7]  Albert A. Rizzo,et al.  Initial Validation of a Virtual Environment for Assessment of Memory Functioning: Virtual Reality Cognitive Performance Assessment Test , 2008, Cyberpsychology Behav. Soc. Netw..

[8]  Thomas D. Parsons,et al.  Virtual Reality Stroop Task for Neurocognitive Assessment , 2011, MMVR.

[9]  S. Urbina.  Essentials of Psychological Testing , 2005, PsyPag Quarterly.

[10]  Susan D. Voyer,et al.  Magnitude of sex differences in spatial abilities: a meta-analysis and consideration of critical variables. , 1995, Psychological bulletin.

[11]  James S. Elkind,et al.  A Simulated Reality Scenario Compared with the Computerized Wisconsin Card Sorting Test: An Analysis of Preliminary Results , 2001, Cyberpsychology Behav. Soc. Netw..

[12]  J. Alvarez,et al.  Executive Function and the Frontal Lobes: A Meta-Analytic Review , 2006, Neuropsychology Review.

[13]  Albert Rizzo,et al.  Is Learning and Memory Different in a Virtual Environment? , 2007, The Clinical neuropsychologist.

[14]  J. Galen Buckwalter,et al.  Sex differences in mental rotation and spatial rotation in a virtual environment , 2004, Neuropsychologia.

[15]  R. Heaton,et al.  The Paced Auditory Serial Addition Task (Pasat): Norms For Age, Education, And Ethnicity , 1998, Assessment.

[16]  Albert A. Rizzo,et al.  Virtual environments for targeting cognitive processes: an overview of projects at the University of Southern California , 1999, HCI.

[17]  E. A. Attree,et al.  Probing Memory and Executive Functions with Virtual Reality: Past and Present Studies , 1998, Cyberpsychology Behav. Soc. Netw..

[18]  Brenda K. Wiederhold,et al.  A Virtual Environment for Investigating Schizophrenic Patients' Characteristics: Assessment of Cognitive and Navigation Ability , 2003, Cyberpsychology Behav. Soc. Netw..

[19]  Grant L Iverson,et al.  Computerized Neuropsychological Assessment Devices: Joint Position Paper of the American Academy of Clinical Neuropsychology and the National Academy of Neuropsychology , 2012, The Clinical neuropsychologist.

[20]  Bruce A. Bracken,et al.  Psychometric Characteristics of Assessment Procedures , 2003 .

[21]  Luciano Gamberini.  Virtual Reality as a New Research Tool for the Study of Human Memory , 2000, Cyberpsychology Behav. Soc. Netw..

[22]  Christopher G. Courtney,et al.  An initial validation of the Virtual Reality Paced Auditory Serial Addition Test in a college sample , 2014, Journal of Neuroscience Methods.

[23]  R. Adams,et al.  Distractibility in Attention/Deficit/ Hyperactivity Disorder (ADHD): The Virtual Reality Classroom , 2009, Child neuropsychology : a journal on normal and abnormal development in childhood and adolescence.

[24]  Noomi Katz,et al.  Validation of the Virtual MET as an assessment tool for executive functions , 2009, Neuropsychological rehabilitation.

[25]  J. Marshall,et al.  VISUOSPATIAL NEGLECT: UNDERLYING FACTORS AND TEST SENSITIVITY , 1989, The Lancet.

[26]  É. Sorita,et al.  Do patients with traumatic brain injury learn a route in the same way in real and virtual environments? , 2013, Disability and rehabilitation.

[27]  D. A. Grant,et al.  A behavioral analysis of degree of reinforcement and ease of shifting to new responses in a Weigl-type card-sorting problem. , 1948, Journal of experimental psychology.

[28]  Martin Ebner,et al.  The XAOS Metric - Understanding Visual Complexity as Measure of Usability , 2010, USAB.

[29]  Christopher G. Courtney,et al.  Virtual reality Stroop task for assessment of supervisory attentional processing , 2013, Journal of clinical and experimental neuropsychology.

[30]  Albert A. Rizzo,et al.  Analysis of assets for virtual reality applications in neuropsychology , 2004 .

[31]  P. Nolin,et al.  Development and initial assessment of a new paradigm for assessing cognitive and motor inhibition: The bimodal virtual-reality Stroop , 2012, Journal of Neuroscience Methods.

[32]  Olejnik,et al.  Measures of Effect Size for Comparative Studies: Applications, Interpretations, and Limitations. , 2000, Contemporary educational psychology.

[33]  Todd Bowerly,et al.  A Virtual Reality Scenario for All Seasons: The Virtual Classroom , 2009, CNS Spectrums.

[34]  L. Hedges,et al.  Introduction to Meta‐Analysis , 2009, International Coaching Psychology Review.

[35]  Pierre Nolin,et al.  Assessment of executive function in adolescence: A comparison of traditional and virtual reality tools , 2013, Journal of Neuroscience Methods.

[36]  Andreas Holzinger,et al.  The effect of previous exposure to technology on acceptance and its importance in usability and accessibility engineering , 2011, Universal Access in the Information Society.

[37]  Albert A. Rizzo,et al.  The Virtual Classroom: A Virtual Reality Environment for the Assessment and Rehabilitation of Attention Deficits , 2000, Cyberpsychology Behav. Soc. Netw..

[38]  A. Rizzo,et al.  Virtual Reality and Neuropsychology: Upgrading the Current Tools , 2002, The Journal of head trauma rehabilitation.

[39]  Stéphane Bouchard,et al.  Assessment of Inhibition Deficits with the Virtual Classroom in Children with Traumatic Brain Injury: A Pilot-Study , 2009, Annual Review of Cybertherapy and Telemedicine.

[40]  Albert A Rizzo,et al.  The Utility of a Continuous Performance Test Embedded in Virtual Reality in Measuring ADHD-Related Deficits , 2009, Journal of developmental and behavioral pediatrics : JDBP.

[41]  Christopher G. Courtney,et al.  Validity of the Virtual Reality Stroop Task (VRST) in active duty military , 2013, Journal of clinical and experimental neuropsychology.

[42]  Howard Rheingold,et al.  Virtual Reality , 1991 .

[43]  Albert A. Rizzo,et al.  Virtual Reality Paced Serial Assessment Test for Neuropsychological Assessment of a Military Cohort , 2012, MMVR.

[44]  J. Stroop.  Studies of interference in serial verbal reactions. , 1992 .

[45]  M Rydmark,et al.  Neglect assessment as an application of virtual reality , 2007, Acta neurologica Scandinavica.

[46]  Albert A. Rizzo,et al.  Comparison of Two VR Platforms for Rehabilitation: Video Capture versus HMD , 2005, Presence: Teleoperators & Virtual Environments.

[47]  James S. Elkind.  Use of Virtual Reality to Diagnose and Habilitate People with Neurological Dysfunctions , 1998, Cyberpsychology Behav. Soc. Netw..

[48]  Martina Ziefle,et al.  Investigating paper vs. screen in real-life hospital workflows: Performance contradicts perceived superiority of paper in the user experience , 2011, Int. J. Hum. Comput. Stud..

[49]  Rodney L. Myers,et al.  Virtual Reality and Left Hemineglect: A Technology for Assessment and Therapy , 2000, Cyberpsychology Behav. Soc. Netw..

[50]  L M Greenberg,et al.  Developmental normative data on the test of variables of attention (T.O.V.A.). , 1993, Journal of child psychology and psychiatry, and allied disciplines.

[51]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[52]  Albert A. Rizzo,et al.  Methylphenidate Effect in Children With ADHD Can Be Measured by an Ecologically Valid Continuous Performance Test Embedded in Virtual Reality , 2010, CNS Spectrums.

[53]  S Duval,et al.  Trim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐Analysis , 2000, Biometrics.