Adapting Visual Complexity Based on Electrodermal Activity Improves Working Memory Performance in Virtual Reality

Biocybernetic loops encompass users' state detection and system adaptation based on physiological signals. Current adaptive systems limit the adaptation to task features such as task difficulty or multitasking demands. However, virtual reality allows the manipulation of task-irrelevant elements in the environment. We present a physiologically adaptive system that adjusts the virtual environment based on physiological arousal, i.e., electrodermal activity. We conducted a user study with our adaptive system in social virtual reality to verify improved performance. Here, participants completed an n-back task, and we adapted the visual complexity of the environment by changing the number of non-player characters. Our results show that an adaptive virtual reality can control users' comfort, performance, and workload by adapting the visual complexity based on physiological arousal. Thus, our physiologically adaptive system improves task performance and perceived workload. Finally, we embed our findings in physiological computing and discuss applications in various scenarios.

[1]  L. Menghini,et al.  SensCon: Embedding Physiological Sensing into Virtual Reality Controllers , 2023, Proc. ACM Hum. Comput. Interact..

[2]  Hai-Ning Liang,et al.  Real-Time Prediction of Simulator Sickness in Virtual Reality Games , 2023, IEEE Transactions on Games.

[3]  Sven Mayer,et al.  Exploring Physiological Correlates of Visual Complexity Adaptation: Insights from EDA, ECG, and EEG Data for Adaptation Evaluation in VR Adaptive Systems , 2023, CHI Extended Abstracts.

[4]  Sven Mayer,et al.  How Can Mixed Reality Benefit From Physiologically-Adaptive Systems? Challenges and Opportunities for Human Factors Applications , 2023, ArXiv.

[5]  Paweł W. Woźniak,et al.  A Survey on Measuring Cognitive Workload in Human-Computer Interaction , 2023, ACM Comput. Surv..

[6]  T. Parsons,et al.  Machine learning classification analysis for an adaptive virtual reality Stroop task , 2023, Virtual Reality.

[7]  I. Gilchrist,et al.  Dynamically changing attention in complex visual stimuli , 2022, Journal of Vision.

[8]  A. Butz,et al.  Designing Mobile MR Workspaces , 2022, Proc. ACM Hum. Comput. Interact..

[9]  Benedikt V. Ehinger,et al.  Adapting visualizations and interfaces to the user , 2022, it Inf. Technol..

[10]  Sven Mayer,et al.  Virtual Reality Adaptation Using Electrodermal Activity to Support the User Experience , 2022, Big Data Cogn. Comput..

[11]  Miguel Melo,et al.  Systematic Review of Comparative Studies of the Impact of Realism in Immersive Virtual Experiences , 2022, ACM Comput. Surv..

[12]  Robin Welsch,et al.  Proxemics for Human-Agent Interaction in Augmented Reality , 2022, CHI.

[13]  T. Kosch,et al.  The Placebo Effect of Artificial Intelligence in Human–Computer Interaction , 2022, ACM Trans. Comput. Hum. Interact..

[14]  Jaewook Lee,et al.  User Preference for Navigation Instructions in Mixed Reality , 2022, 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[15]  Eric Tatt Wei Ho,et al.  Phasic Electrodermal Activity Indicates Changes in Workload and Affective States , 2021, 2021 International Conference on Intelligent Cybernetics Technology & Applications (ICICyTA).

[16]  Matthew S. Goodwin,et al.  Adaptive thresholding increases ability to detect changes in rate of skin conductance responses to psychologically arousing stimuli , 2021 .

[17]  Günter Daniel Rey,et al.  Is a Preference for Realism Really Naive After All? A Cognitive Model of Learning with Realistic Visualizations , 2021, Educational Psychology Review.

[18]  Mercedes Valmisa Adapting , 2021 .

[19]  Prasad Calyam,et al.  Rule-based Adaptations to Control Cybersickness in Social Virtual Reality Learning Environments , 2021, 2021 8th International Conference on Future Internet of Things and Cloud (FiCloud).

[20]  Eric Redlinger,et al.  Enhanced Cognitive Training using Virtual Reality: Examining a Memory Task Modified for Use in Virtual Environments , 2021, AIVR.

[21]  Tilman Dingler,et al.  A Critique of Electrodermal Activity Practices at CHI , 2021, CHI.

[22]  Yizhen Huang,et al.  Classroom complexity affects student teachers' behavior in a VR classroom , 2021, Comput. Educ..

[23]  Andreas Butz,et al.  Rear-Seat Productivity in Virtual Reality: Investigating VR Interaction in the Confined Space of a Car , 2021, Multimodal Technol. Interact..

[24]  Carryl L. Baldwin,et al.  Grand Field Challenges for Cognitive Neuroergonomics in the Coming Decade , 2021, Frontiers in Neuroergonomics.

[25]  G. Borghini,et al.  The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence , 2021, Scientific Reports.

[26]  Gongfa Li,et al.  Crowd emotion evaluation based on fuzzy inference of arousal and valence , 2021, Neurocomputing.

[27]  Guo Freeman,et al.  Body, Avatar, and Me , 2021, Proc. ACM Hum. Comput. Interact..

[28]  Jarrod A. Lewis-Peacock,et al.  Distraction in Visual Working Memory: Resistance is Not Futile , 2021, Trends in Cognitive Sciences.

[29]  Christos Mousas,et al.  A Virtual Reality Framework for Human-Virtual Crowd Interaction Studies , 2020, 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR).

[30]  Marc Erich Latoschik,et al.  Latency and Cybersickness: Impact, Causes, and Measures. A Review , 2020, Frontiers in Virtual Reality.

[31]  M. Masuch,et al.  The Experience of Social Touch in Multi-User Virtual Reality , 2020, VRST.

[32]  Nicolas Martin,et al.  Virtual reality sickness detection: an approach based on physiological signals and machine learning , 2020, 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[33]  Walt Detmar Meurers,et al.  Toward neuroadaptive support technologies for improving digital reading: a passive BCI-based assessment of mental workload imposed by text difficulty and presentation speed during reading , 2020, User Modeling and User-Adapted Interaction.

[34]  Jan C. Brammer,et al.  NeuroKit2: A Python toolbox for neurophysiological signal processing , 2020, Behavior Research Methods.

[35]  Francesco Piccione,et al.  Theta and alpha oscillations as signatures of internal and external attention to delayed intentions: A magnetoencephalography (MEG) study , 2020, NeuroImage.

[36]  M. Baldo,et al.  Responses to auditory distractors during a class depend on participant’s enjoyment: An eye-tracking and skin conductance study. , 2019 .

[37]  Iztok Humar,et al.  Virtual Reality Sickness and Challenges Behind Different Technology and Content Settings , 2019, Mobile Networks and Applications.

[38]  Francesca De Crescenzio,et al.  EEG Alpha Power Is Modulated by Attentional Changes during Cognitive Tasks and Virtual Reality Immersion , 2019, Comput. Intell. Neurosci..

[39]  Albrecht Schmidt,et al.  Your skin resists: exploring electrodermal activity as workload indicator during manual assembly , 2019, EICS.

[40]  Beata Starzyńska,et al.  Cyber Sickness in Industrial Virtual Reality Training , 2019, Lecture Notes in Mechanical Engineering.

[41]  Arindam Dey,et al.  Exploration of an EEG-Based Cognitively Adaptive Training System in Virtual Reality , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[42]  N. Intrator,et al.  Electrical fingerprint of the amygdala guides neurofeedback training for stress resilience , 2018, Nature Human Behaviour.

[43]  Peta Wyeth,et al.  Don't Sweat the Small Stuff: The Effect of Challenge-Skill Manipulation on Electrodermal Activity , 2018, CHI PLAY.

[44]  Simon Farrell,et al.  Benchmarks for models of short-term and working memory. , 2018, Psychological bulletin.

[45]  Stephen H Fairclough,et al.  FNIRS activity in the prefrontal cortex and motivational intensity: impact of working memory load, financial reward, and correlation-based signal improvement , 2018, Neurophotonics.

[46]  Tilman Deuschel,et al.  On the Influence of Human Factors in Adaptive User Interface Design , 2018, UMAP.

[47]  David J. Zielinski,et al.  Measuring visual search and distraction in immersive virtual reality , 2018, Royal Society Open Science.

[48]  Chenglei Yang,et al.  Exploring the Weak Association between Flow Experience and Performance in Virtual Environments , 2018, CHI.

[49]  Maic Masuch,et al.  Increasing Presence in a Mixed Reality Application by Integrating a Real Time Tracked Full Body Representation , 2017, ACE.

[50]  Richard E. Mayer,et al.  Adding immersive virtual reality to a science lab simulation causes more presence but less learning , 2017, Learning and Instruction.

[51]  Eugene Nalivaiko,et al.  Effects of visual flow direction on signs and symptoms of cybersickness , 2017, PloS one.

[52]  Markus Funk,et al.  ABBAS: An Adaptive Bio-sensors Based Assistive System , 2017, CHI Extended Abstracts.

[53]  A. Baddeley,et al.  Executive and Perceptual Distraction in Visual Working Memory , 2017, Journal of experimental psychology. Human perception and performance.

[54]  Y. Yeshurun,et al.  Large inter-individual and intra-individual variability in the effect of perceptual load , 2017, PloS one.

[55]  Alireza Mazloumi Gavgani,et al.  Profiling subjective symptoms and autonomic changes associated with cybersickness , 2017, Autonomic Neuroscience.

[56]  Ryohei Egusa,et al.  EDA-BASED ESTIMATION OF VISUAL ATTENTION BY OBSERVATION OF EYE BLINK FREQUENCY , 2017 .

[57]  Matthias Rauterberg,et al.  Handbook of Digital Games and Entertainment Technologies , 2016 .

[58]  Stephen H. Fairclough,et al.  Evaluation of an Adaptive Game that Uses EEG Measures Validated during the Design Process as Inputs to a Biocybernetic Loop , 2016, Front. Hum. Neurosci..

[59]  B. Dyson,et al.  Cross-modal perceptual load: the impact of modality and individual differences , 2016, Experimental Brain Research.

[60]  Sabarish V. Babu,et al.  Effects of Virtual Human Appearance Fidelity on Emotion Contagion in Affective Inter-Personal Simulations , 2016, IEEE Transactions on Visualization and Computer Graphics.

[61]  Charalambos Poullis,et al.  Psychophysiological responses to virtual crowds: Implications for wearable computing , 2015, 2015 International Conference on Affective Computing and Intelligent Interaction (ACII).

[62]  Akane Sano,et al.  Wavelet-based motion artifact removal for electrodermal activity , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[63]  Eric D. Ragan,et al.  Effects of Field of View and Visual Complexity on Virtual Reality Training Effectiveness for a Visual Scanning Task , 2015, IEEE Transactions on Visualization and Computer Graphics.

[64]  Roderick Murray-Smith,et al.  A Dose of Reality: Overcoming Usability Challenges in VR Head-Mounted Displays , 2015, CHI Extended Abstracts.

[65]  R. Gutierrez-Osuna,et al.  A comparative study of game mechanics and control laws for an adaptive physiological game , 2015, Journal on Multimodal User Interfaces.

[66]  Daniel Afergan,et al.  Dynamic difficulty using brain metrics of workload , 2014, CHI.

[67]  Fiona McNab,et al.  Dissociating Distractor-Filtering at Encoding and During Maintenance , 2014, Journal of experimental psychology. Human perception and performance.

[68]  H. Meij,et al.  Game Experience Questionnaire , 2014 .

[69]  Peter Carruthers,et al.  Evolution of working memory , 2013, Proceedings of the National Academy of Sciences.

[70]  Thomas D. Parsons,et al.  Adaptive virtual environments for neuropsychological assessment in serious games , 2012, IEEE Transactions on Consumer Electronics.

[71]  Brian C. Horslen,et al.  Influence of real and virtual heights on standing balance. , 2012, Gait & posture.

[72]  Marko Munih,et al.  A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing , 2012, Interact. Comput..

[73]  M. Wenger,et al.  Processing capacity under perceptual and cognitive load: a closer look at load theory. , 2011, Journal of experimental psychology. Human perception and performance.

[74]  E. Vogel,et al.  Individual Differences in Recovery Time From Attentional Capture , 2011, Psychological science.

[75]  Mel Slater,et al.  Proxemics with multiple dynamic characters in an immersive virtual environment , 2010, TAP.

[76]  M. Benedek,et al.  Decomposition of skin conductance data by means of nonnegative deconvolution , 2010, Psychophysiology.

[77]  Brent Lance,et al.  Optimal Arousal Identification and Classification for Affective Computing Using Physiological Signals: Virtual Reality Stroop Task , 2010, IEEE Transactions on Affective Computing.

[78]  M. Benedek,et al.  A continuous measure of phasic electrodermal activity , 2010, Journal of Neuroscience Methods.

[79]  Susanne M. Jaeggi,et al.  The concurrent validity of the N-back task as a working memory measure , 2010, Memory.

[80]  Hing Yee Eng,et al.  Visual working memory for simple and complex visual stimuli. , 2010, Psychonomic bulletin & review.

[81]  Alan Chalmers,et al.  Levels of realism: from virtual reality to real virtuality , 2008, SCCG.

[82]  R. Wright,et al.  Refining the Prediction of Effort: Brehm's Distinction between Potential Motivation and Motivation Intensity , 2008 .

[83]  Robert J Barry,et al.  Arousal and activation effects on physiological and behavioral responding during a continuous performance task. , 2007, Acta neurobiologiae experimentalis.

[84]  J. Lagopoulos Electrodermal activity , 2007, Acta Neuropsychiatrica.

[85]  Yuanzhen Li,et al.  Measuring visual clutter. , 2007, Journal of vision.

[86]  D. J. McKeefry,et al.  Speed selectivity in visual short term memory for motion , 2007, Vision Research.

[87]  Wolfram Boucsein,et al.  Combining Skin Conductance and Heart Rate Variability for Adaptive Automation During Simulated IFR Flight , 2007, HCI.

[88]  Roy Stripling,et al.  Automated SAF Adaptation Tool (ASAT) , 2007, HCI.

[89]  Kathryn M. McMillan,et al.  Self-paced working memory: Validation of verbal variations of the n-back paradigm , 2007, Brain Research.

[90]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[91]  Clayton E. Curtis,et al.  Differential effects of distraction during working memory on delay-period activity in the prefrontal cortex and the visual association cortex , 2006, NeuroImage.

[92]  S. Fairclough,et al.  Prediction of subjective states from psychophysiology: A multivariate approach , 2006, Biological Psychology.

[93]  J. Gross,et al.  Mechanisms of Virtual Reality Exposure Therapy: The Role of the Behavioral Activation and Behavioral Inhibition Systems , 2005, Applied psychophysiology and biofeedback.

[94]  Andrea C. Pierno,et al.  Effects of increasing visual load on aurally and visually guided target acquisition in a virtual environment. , 2005, Applied ergonomics.

[95]  P. Cavanagh,et al.  The Capacity of Visual Short-Term Memory is Set Both by Visual Information Load and by Number of Objects , 2004, Psychological science.

[96]  N. Lavie,et al.  The Role of Perceptual Load in Processing Distractor Faces , 2003, Psychological science.

[97]  James L. Szalma,et al.  The future of neuroergonomics , 2003 .

[98]  Henry Been-Lirn Duh,et al.  Effects of field of view on presence, enjoyment, memory, and simulator sickness in a virtual environment , 2002, Proceedings IEEE Virtual Reality 2002.

[99]  P. Pauli,et al.  Repeated exposure of flight phobics to flights in virtual reality. , 2001, Behaviour research and therapy.

[100]  C. Frith,et al.  The Role of Working Memory in Visual Selective Attention , 2001, Science.

[101]  E Gordon,et al.  The relationship between quantified EEG and skin conductance level. , 1996, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[102]  N. Lavie Perceptual load as a necessary condition for selective attention. , 1995, Journal of experimental psychology. Human perception and performance.

[103]  A. Pope,et al.  Biocybernetic system evaluates indices of operator engagement in automated task , 1995, Biological Psychology.

[104]  N Morris,et al.  Characteristics of visual interference with visuospatial working memory. , 1994, British journal of psychology.

[105]  Michael E. McCauley,et al.  Cybersickness: Perception of Self-Motion in Virtual Environments , 1992, Presence: Teleoperators & Virtual Environments.

[106]  Mark W. Greenlee,et al.  Stimulus-specific mechanisms of visual short-term memory , 1991, Vision Research.

[107]  J. Duncan,et al.  Visual search and stimulus similarity. , 1989, Psychological review.

[108]  W. Boucsein,et al.  Dependence of skin conductance reactions and skin resistance reactions upon previous level. , 1984, Psychophysiology.

[109]  G. Ben-Shakhar,et al.  Publication recommendations for electrodermal measurements. , 1981, Psychophysiology.

[110]  D. Groggel Practical Nonparametric Statistics , 1972, Technometrics.

[111]  R. D'Agostino An omnibus test of normality for moderate and large size samples , 1971 .

[112]  Florian Alt,et al.  When Friends Become Strangers: Understanding the Influence of Avatar Gender on Interpersonal Distance in Virtual Reality , 2021, INTERACT.

[113]  Abdul Momin,et al.  Analysis of Electrodermal Activity Signal Collected During Visual Attention Oriented Tasks , 2019, IEEE Access.

[114]  Stephen H. Fairclough,et al.  Physiological Computing and Intelligent Adaptation , 2017 .

[115]  Michael Richter,et al.  Three Decades of Research on Motivational Intensity Theory: What We Have Learned About Effort and What We Still Don't Know , 2016 .

[116]  E. Scilingo,et al.  Modeling for the Analysis of the EDA , 2016 .

[117]  Doron Friedman,et al.  Brain-Computer Interfacing and Virtual Reality , 2015 .

[118]  Stephen H. Fairclough,et al.  Fundamentals of physiological computing , 2009, Interact. Comput..

[119]  J. Dusek,et al.  Impact of Incremental Increases in Cognitive Workload on Physiological Arousal and Performance in Young Adult Drivers , 2009 .

[120]  Iwan Kartiko,et al.  Effects of digital content on motion sickness in immersive virtual environments , 2008 .

[121]  Jordan DeLong,et al.  Predictions of Performance by EEG and Skin Conductance , 2007 .

[122]  Michael L. Mack,et al.  Identifying the Perceptual Dimensions of Visual Complexity of Scenes , 2004 .

[123]  Heikki Lyytinen,et al.  The Orienting Response in Information Processing , 2002 .

[124]  Robert H. Logie,et al.  Working Memory, Mental Capacity and Science Education: towards an understanding of the ‘working memory overload hypothesis’ , 1993 .

[125]  R. Nikula Psychological correlates of nonspecific skin conductance responses. , 1991, Psychophysiology.

[126]  J. Desor,et al.  Toward a Psychological Theory of Crowding. , 1972 .