Measuring Cognitive Load and Insight: A Methodology Exemplified in a Virtual Reality Learning Context

Recent improvements of Virtual Reality (VR) technology have enabled researchers to investigate the benefits VR may provide for various domains such as health, entertainment, training, and education. A significant proportion of VR system evaluations rely on perception-based measures such as user pre-and post-questionnaires and interviews. While these self-reports provide valuable insights into users' perceptions of VR environments, recent developments in digital sensors and data collection techniques afford researchers access to measures of physiological response. This work explores the merits of physiological measures in the evaluation of emotional responses in virtual environments (ERVE). We include and place at the center of our ERVE methodology emotional response data by way of electrodermal activity and heart-rate detection which are analyzed in conjunction with event-driven data to derive further measures. In this paper, we present our ERVE methodology together with a case study within the context of VR-based learning in which we derive measures of cognitive load and moments of insight. We discuss our methodology, and its potential for use in many other application and research domains to provide more in-depth and objective analyses of experiences within VR.

[1]  K. Ottenbacher,et al.  Task performance in virtual environments used for cognitive rehabilitation after traumatic brain injury. , 1998, Archives of physical medicine and rehabilitation.

[2]  Luca Citi,et al.  cvxEDA: A Convex Optimization Approach to Electrodermal Activity Processing , 2016, IEEE Transactions on Biomedical Engineering.

[3]  J. Bhattacharya,et al.  Deconstructing Insight: EEG Correlates of Insightful Problem Solving , 2008, PloS one.

[4]  Rosalind W. Picard,et al.  A Wearable Sensor for Unobtrusive, Long-Term Assessment of Electrodermal Activity , 2010, IEEE Transactions on Biomedical Engineering.

[5]  Zahira Merchant,et al.  Effectiveness of virtual reality-based instruction on students' learning outcomes in K-12 and higher education: A meta-analysis , 2014, Comput. Educ..

[6]  Doug A. Bowman,et al.  Virtual Reality: How Much Immersion Is Enough? , 2007, Computer.

[7]  Springer-Verlag London Limited Monitoring of mental workload levels during an everyday life office-work scenario , 2013 .

[8]  M. Ott,et al.  ON IMMERSIVE VIRTUAL REALITY IN EDUCATION : STATE OF THE ART AND PERSPECTIVES , 2015 .

[9]  Holger Regenbrecht,et al.  Mild stress stimuli built into a non-immersive virtual environment can elicit actual stress responses , 2017, Behav. Inf. Technol..

[10]  S. Khalfa,et al.  Event-related skin conductance responses to musical emotions in humans , 2002, Neuroscience Letters.

[11]  Mel Slater,et al.  Variations in physiological responses of participants during different stages of an immersive virtual environment experiment , 2006, VRST '06.

[12]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[13]  Jonathan Steuer,et al.  Defining virtual reality: dimensions determining telepresence , 1992 .

[14]  Mel Slater,et al.  An Analysis of Eye Scanpath Entropy in a Progressively Forming Virtual Environment , 2009, PRESENCE: Teleoperators and Virtual Environments.

[15]  Adrian Basarab,et al.  Towards an automatic early stress recognition system for office environments based on multimodal measurements: A review , 2016, J. Biomed. Informatics.

[16]  Sehat Ullah,et al.  Interactive Virtual Chemistry Laboratory for Simulation of High School Experiments , 2014 .

[17]  Chris Barker,et al.  An Experimental Study on Fear of Public Speaking Using a Virtual Environment , 2006, Cyberpsychology Behav. Soc. Netw..

[18]  Giuseppe Riva,et al.  Panic and Agoraphobia in a Virtual World , 2002, Cyberpsychology Behav. Soc. Netw..

[19]  Dong Pyo Jang,et al.  Physiological Monitoring as an Objective Tool in Virtual Reality Therapy , 2002, Cyberpsychology Behav. Soc. Netw..

[20]  Robert S. Kennedy,et al.  Simulator Sickness Questionnaire: An enhanced method for quantifying simulator sickness. , 1993 .

[21]  Nikos Papadopoulos,et al.  Full immersive virtual environment CAVETM in chemistry education , 2008, Comput. Educ..

[22]  Sharif Razzaque,et al.  The Responses of People to Virtual Humans in an Immersive Virtual Environment , 2005, Presence: Teleoperators & Virtual Environments.

[23]  Khe Foon Hew,et al.  Use of three-dimensional (3-D) immersive virtual worlds in K-12 and higher education settings: A review of the research , 2010, Br. J. Educ. Technol..

[24]  J. Kounios,et al.  The Aha! Moment , 2009 .

[25]  Michael Meehan,et al.  Physiological measures of presence in stressful virtual environments , 2002, SIGGRAPH.

[26]  Mel Slater,et al.  The Influence of Dynamic Shadows on Presence in Immersive Virtual Environments , 1995, Virtual Environments.

[27]  Hyun-Sup Kim,et al.  Application of virtual reality technology in biology education , 2003 .

[28]  Julian Frommel,et al.  Integrated Questionnaires: Maintaining Presence in Game Environments for Self-Reported Data Acquisition , 2015, CHI PLAY.

[29]  Brenda K. Wiederhold,et al.  Fear of Flying: A Case Report Using Virtual Reality Therapy with Physiological Monitoring , 1998, Cyberpsychology Behav. Soc. Netw..

[30]  Mel Slater,et al.  Visual Realism Enhances Realistic Response in an Immersive Virtual Environment , 2009, IEEE Computer Graphics and Applications.

[31]  M. Lombard,et al.  Measuring Presence: The Temple Presence Inventory , 2009 .

[32]  M. Whitton,et al.  Review of Four Studies on the Use of Physiological Reaction as a Measure of Presence in StressfulVirtual Environments , 2005, Applied psychophysiology and biofeedback.

[33]  Regan L. Mandryk,et al.  A continuous and objective evaluation of emotional experience with interactive play environments , 2006, CHI.

[34]  Hunter G. Hoffman,et al.  Modulation of thermal pain-related brain activity with virtual reality: evidence from fMRI , 2004, Neuroreport.

[35]  Bin Shyan Jong,et al.  A Web-based virtual reality physics laboratory , 2003, Proceedings 3rd IEEE International Conference on Advanced Technologies.

[36]  Marcus Vollmer,et al.  A robust, simple and reliable measure of heart rate variability using relative RR intervals , 2015, 2015 Computing in Cardiology Conference (CinC).

[37]  Maria V. Sanchez-Vives,et al.  From presence to consciousness through virtual reality , 2005, Nature Reviews Neuroscience.

[38]  K. S. Song,et al.  A virtual reality application for geometry classes , 2002, J. Comput. Assist. Learn..

[39]  Mikhail Morozov,et al.  Virtual chemistry laboratory for school education , 2004, IEEE International Conference on Advanced Learning Technologies, 2004. Proceedings..

[40]  Wijnand A. IJsselsteijn,et al.  Presence: concept, determinants, and measurement , 2000, Electronic Imaging.

[41]  Andreas Mühlberger,et al.  Temporal dynamics in the relation between presence and fear in virtual reality , 2015, Comput. Hum. Behav..

[42]  Gert Pfurtscheller,et al.  Heart-Rate Variability and Event-Related ECG in Virtual Environments , 2004 .

[43]  Mel Slater,et al.  How Colorful Was Your Day? Why Questionnaires Cannot Assess Presence in Virtual Environments , 2004, Presence: Teleoperators & Virtual Environments.

[44]  Helmut Hlavacs,et al.  Afraid to Be There? Evaluating the Relation Between Presence, Self-Reported Anxiety, and Heart Rate in a Virtual Public Speaking Task , 2014, Cyberpsychology Behav. Soc. Netw..

[45]  Stellan Ohlsson,et al.  Deep Learning - How the Mind Overrides Experience , 2011 .

[46]  Manolya Kavakli,et al.  Behavioral Presence Test in Threatening Virtual Environments , 2012, PRESENCE: Teleoperators and Virtual Environments.

[47]  Henrik M. Peperkorn,et al.  The impact of perception and presence on emotional reactions: a review of research in virtual reality , 2015, Front. Psychol..

[48]  Franco Tecchia,et al.  Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks , 2015, Interact. Learn. Environ..

[49]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[50]  Dieter Schmalstieg,et al.  Construct3D: A Virtual Reality Application for Mathematics and Geometry Education , 2000, Education and Information Technologies.

[51]  Helmut Hlavacs,et al.  Is virtual reality emotionally arousing? Investigating five emotion inducing virtual park scenarios , 2015, Int. J. Hum. Comput. Stud..

[52]  Rolf Reber,et al.  Gaining Insight Into the “Aha” Experience , 2010 .

[53]  M. Benedek,et al.  A continuous measure of phasic electrodermal activity , 2010, Journal of Neuroscience Methods.

[54]  H. Critchley Review: Electrodermal Responses: What Happens in the Brain , 2002 .

[55]  Gert Pfurtscheller,et al.  Analysis of Physiological Responses to a Social Situation in an Immersive Virtual Environment , 2006, PRESENCE: Teleoperators and Virtual Environments.

[56]  Mary Corbett,et al.  SUMI: the Software Usability Measurement Inventory , 1993, Br. J. Educ. Technol..

[57]  Alistair G. Sutcliffe,et al.  Evaluating the usability of virtual reality user interfaces , 2000, Behav. Inf. Technol..

[58]  Cem Ersoy,et al.  Continuous Stress Detection Using Wearable Sensors in Real Life: Algorithmic Programming Contest Case Study , 2019, Sensors.

[59]  Holger Regenbrecht,et al.  The Experience of Presence: Factor Analytic Insights , 2001, Presence: Teleoperators & Virtual Environments.

[60]  Amory H. Danek,et al.  What about False Insights? Deconstructing the Aha! Experience along Its Multiple Dimensions for Correct and Incorrect Solutions Separately , 2017, Front. Psychol..

[61]  John G. Casali,et al.  A Validated Rating Scale for Global Mental Workload Measurement Applications , 1983 .

[62]  R. Weisberg Toward an integrated theory of insight in problem solving , 2015 .

[63]  Akane Sano,et al.  Automatic identification of artifacts in electrodermal activity data , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[64]  P. Pauli,et al.  Virtual reality exposure in anxiety disorders: Impact on psychophysiological reactivity , 2014, The world journal of biological psychiatry : the official journal of the World Federation of Societies of Biological Psychiatry.

[65]  Martin Schrepp,et al.  Construction and Evaluation of a User Experience Questionnaire , 2008, USAB.

[66]  Holger Regenbrecht,et al.  Measuring the Sense of Presence and its Relations to Fear of Heights in Virtual Environments , 1998, Int. J. Hum. Comput. Interact..

[67]  Li-Wei Ko,et al.  EEG-Based Assessment of Driver Cognitive Responses in a Dynamic Virtual-Reality Driving Environment , 2007, IEEE Transactions on Biomedical Engineering.

[68]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[69]  Joseph S. Dumas,et al.  Usability assessment methods , 2006 .

[70]  R. Benjamin Knapp,et al.  Affective feedback in a virtual reality based intelligent supermarket , 2017, UbiComp/ISWC Adjunct.

[71]  Koun-Tem Sun,et al.  Research on the application of virtual reality on arts core curricula , 2010, 5th International Conference on Computer Sciences and Convergence Information Technology.

[72]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[73]  José Manuel Pastor,et al.  Electrodermal Activity Sensor for Classification of Calm/Distress Condition , 2017, Sensors.