Estimating mental workload through event-related fluctuations of pupil area during a task in a virtual world.

Monitoring mental load for optimal performance has become increasingly central with the recently evolving need to cope with exponentially increasing amounts of data. This paper describes a non-intrusive, objective method to estimate mental workload in an immersive virtual reality system, through analysis of frequencies of pupil fluctuations. We tested changes in mental workload with a number of task-repetitions, level of predictability of the task and the effect of prior experience in predictable task performance, on mental workload of unpredictable task performance. Two measures were used to calculate mental workload: the ratio of Low Frequency to High Frequency components of pupil fluctuations, and the High Frequency alone, all extracted from the Power Spectrum Density of pupil fluctuations. Results show that mental workload decreases with a number of repetitions, creating a mode in which the brain acts as an automatic controller. Automaticity during training occurs only after a minimal number of repetitions, which once achieved, resulted in further improvements in the performance of unpredictable motor tasks, following training in a predictable task. These results indicate that automaticity is a central component in the transfer of skills from highly predictable to low predictable motor tasks. Our results suggest a potentially applicable method to brain-computer-interface systems that adapt to human mental workload, and provide intelligent automated support for enhanced performance.

[1]  A. Murata,et al.  Evaluation of mental workload by fluctuation analysis of pupil area , 1998, Proceedings of the 20th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Vol.20 Biomedical Engineering Towards the Year 2000 and Beyond (Cat. No.98CH36286).

[2]  Siyuan Chen,et al.  Eye activity as a measure of human mental effort in HCI , 2011, IUI '11.

[3]  Miriam Reiner,et al.  The role of haptics in immersive telecommunication environments , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[4]  P A Hancock,et al.  Physiological reflections of mental workload. , 1985, Aviation, space, and environmental medicine.

[5]  P. Borgdorff,et al.  Respiratory fluctuations in pupil size. , 1975, The American journal of physiology.

[6]  John L. Sibert,et al.  Heart rate variability: indicator of user state as an aid to human-computer interaction , 1998, CHI.

[7]  Robert Oostenveld,et al.  Estimating workload using EEG spectral power and ERPs in the n-back task , 2012, Journal of neural engineering.

[8]  R. Meulenbroek,et al.  Stimulus–response compatibility and affective computing: a review , 2007 .

[9]  Xiaolu Dong,et al.  Mental workload measurement for emergency operating procedures in digital nuclear power plants , 2013, Ergonomics.

[10]  Brian P. Bailey,et al.  Categories & Subject Descriptors: H.5.2 [Information , 2022 .

[11]  Miriam Reiner,et al.  The Virtual Hand Illusion and Body Ownership , 2008, EuroHaptics.

[12]  Atsuo Murata,et al.  Proposal of Estimation Method of Stable Fixation Points for Eye-gaze Input Interface , 2013, HCI.

[13]  Leslie G. Ungerleider,et al.  Brain plasticity related to the consolidation of motor sequence learning and motor adaptation , 2010, Proceedings of the National Academy of Sciences.

[14]  Mickaël Causse,et al.  Monitoring Cognitive and Emotional Processes Through Pupil and Cardiac Response During Dynamic Versus Logical Task , 2010, Applied psychophysiology and biofeedback.

[15]  S. Miyake,et al.  Power spectral analysis of heart rate variability in type A females during a psychomotor task. , 1998, Journal of psychosomatic research.

[16]  M. Mon-Williams,et al.  Motor Control and Learning , 2006 .

[17]  Samara L. Firebaugh,et al.  Cognitive stress recognition , 2013, 2013 IEEE International Instrumentation and Measurement Technology Conference (I2MTC).

[18]  Megan H. Papesh,et al.  Pupil Dilation Reflects the Creation and Retrieval of Memories , 2012 .

[19]  E. Hess,et al.  Pupil Size as Related to Interest Value of Visual Stimuli , 1960, Science.

[20]  J. Doyon,et al.  Reorganization and plasticity in the adult brain during learning of motor skills , 2005, Current Opinion in Neurobiology.

[21]  Atsuo Murata,et al.  Evaluation of Mental Workload by Variability of Pupil Area , 2000 .

[22]  Dov Sagi,et al.  Global resistance to local perceptual adaptation in texture discrimination , 2009, Vision Research.

[23]  Blake Hannaford,et al.  Commentary: Virtual reality and robotics in neurosurgery. , 2013, Neurosurgery.

[24]  Y. Dudai,et al.  Rites of Passage of the Engram Reconsolidation and the Lingering Consolidation Hypothesis , 2004, Neuron.

[25]  Warren D. Smith,et al.  An ergonomic comparison of robotic and laparoscopic technique: the influence of surgeon experience and task complexity. , 2003, The Journal of surgical research.

[26]  Joachim Vogt,et al.  The Impact of Workload on Heart Rate and Blood Pressure in En-Route and Tower Air Traffic Control , 2006 .

[27]  J. Jolles,et al.  Pupil dilation in response preparation. , 2008, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[28]  B. Tversky,et al.  Effects of visual and verbal presentation on cognitive load in vigilance, memory, and arithmetic tasks. , 2011, Psychophysiology.

[29]  Antonio Frisoli,et al.  Understanding and Realizing Presence in the Presenccia Project , 2007, IEEE Computer Graphics and Applications.

[30]  A. Walden,et al.  Spectral analysis for physical applications : multitaper and conventional univariate techniques , 1996 .

[31]  C. Neuper,et al.  Combining Brain–Computer Interfaces and Assistive Technologies: State-of-the-Art and Challenges , 2010, Front. Neurosci..

[32]  G Schneider,et al.  Eye tracking for assessment of workload: a pilot study in an anaesthesia simulator environment. , 2011, British journal of anaesthesia.

[33]  José Manuel Andújar Márquez,et al.  Interaction and evaluation of an augmented virtuality assistance system for teleoperated robots , 2012, 2012 IEEE International Symposium on Robotic and Sensors Environments Proceedings.

[34]  Maria V. Sanchez-Vives,et al.  Virtual Hand Illusion Induced by Visuomotor Correlations , 2010, PloS one.

[35]  Mel Slater,et al.  Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[36]  G. Calcagnini,et al.  of the 23 rd Annual EMBS International Conference , October 25-28 , Istanbul , Turkey BARORECEPTOR-SENSITIVE FLUCTUATIONS OF HEART RATE AND PUPIL DIAMETER , 2004 .

[37]  Katherine M. Tsui,et al.  Robots in the loop: Telepresence robots in everyday life , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[38]  Veikko Surakka,et al.  Pupil size variation as an indication of affective processing , 2003, Int. J. Hum. Comput. Stud..

[39]  R. Passingham,et al.  That's My Hand! Activity in Premotor Cortex Reflects Feeling of Ownership of a Limb , 2004, Science.

[40]  Jan-Michael Frahm,et al.  Room-sized informal telepresence system , 2012, 2012 IEEE Virtual Reality Workshops (VRW).

[41]  M. A. Recarte,et al.  Mental workload while driving: effects on visual search, discrimination, and decision making. , 2003, Journal of experimental psychology. Applied.

[42]  M. Fahle Perceptual learning: a case for early selection. , 2004, Journal of vision.

[43]  G. Breithardt,et al.  Heart rate variability: standards of measurement, physiological interpretation and clinical use. Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology. , 1996 .

[44]  Jonathan D. Cohen,et al.  Rubber hands ‘feel’ touch that eyes see , 1998, Nature.

[45]  A. Malliani,et al.  Heart rate variability. Standards of measurement, physiological interpretation, and clinical use , 1996 .

[46]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[47]  Mickaël Causse,et al.  Emotion induction through virtual avatars and its impact on reasoning: evidence from autonomous nervous system measurements and cognitive assessment , 2007 .

[48]  K. J. Vicente,et al.  Spectral Analysis of Sinus Arrhythmia: A Measure of Mental Effort , 1987, Human factors.

[49]  K Yana,et al.  Time-Varying Properties of Respiratory Fluctuations in Pupil Diameter of Human Eyes , 1994, Methods of Information in Medicine.

[50]  Louise Venables,et al.  The influence of task demand and learning on the psychophysiological response. , 2005, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[51]  M. Reiner,et al.  Sensory dominance in combinations of audio, visual and haptic stimuli , 2009, Experimental Brain Research.

[52]  H. Kawasaki,et al.  Respiratory fluctuations of the human pupil , 2004, Experimental Brain Research.

[53]  Brian P. Bailey,et al.  Towards an index of opportunity: understanding changes in mental workload during task execution , 2004, CHI.

[54]  Mehran Anvari,et al.  The impact of latency on surgical precision and task completion during robotic-assisted remote telepresence surgery , 2005, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[55]  Miriam Reiner,et al.  Behavioral Indications of Object-Presence in Haptic Virtual Environments , 2009, Cyberpsychology Behav. Soc. Netw..

[56]  M. Bradley,et al.  The pupil as a measure of emotional arousal and autonomic activation. , 2008, Psychophysiology.

[57]  Miriam Reiner,et al.  Multimodal Virtual Environments: Response Times, Attention, and Presence , 2006, PRESENCE: Teleoperators and Virtual Environments.

[58]  James C. Christensen,et al.  The effects of day-to-day variability of physiological data on operator functional state classification , 2012, NeuroImage.

[59]  José del R. Millán,et al.  Improving Human Performance in a Real Operating Environment through Real-Time Mental Workload Detection , 2007 .

[60]  Anselmo Lastra,et al.  New Generation of Instrumented Ranges: Enabling Automated Performance Analysis , 2009 .