Continuous Mental Effort Evaluation During 3D Object Manipulation Tasks Based on Brain and Physiological Signals

Designing 3D User Interfaces (UI) requires adequate evaluation tools to ensure good usability and user experience. While many evaluation tools are already available and widely used, existing approaches generally cannot provide continuous and objective measures of usability qualities during interaction without interrupting the user. In this paper, we propose to use brain (with ElectroEncephaloGraphy) and physiological (ElectroCardioGraphy, Galvanic Skin Response) signals to continuously assess the mental effort made by the user to perform 3D object manipulation tasks. We first show how this mental effort (a.k.a., mental workload) can be estimated from such signals, and then measure it on 8 participants during an actual 3D object manipulation task with an input device known as the CubTile. Our results suggest that monitoring workload enables us to continuously assess the 3DUI and/or interaction technique ease-of-use. Overall, this suggests that this new measure could become a useful addition to the repertoire of available evaluation tools, enabling a finer grain assessment of the ergonomic qualities of a given 3D user interface.

[1]  Christian Mühl,et al.  Review of the Use of Electroencephalography as an Evaluation Method for Human-Computer Interaction , 2013, PhyCS.

[2]  J. Matias,et al.  Review on psychophysiological methods in game research , 2010 .

[3]  Christian Mühl,et al.  EEG-based workload estimation across affective contexts , 2014, Front. Neurosci..

[4]  Sriram Subramanian,et al.  Comparison of User Performance in Mixed 2D-3D Multi-Display Environments , 2013, INTERACT.

[5]  Clemens Brunner,et al.  Better than random? A closer look on BCI results , 2008 .

[6]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[7]  J. Wolpaw,et al.  EMG contamination of EEG: spectral and topographical characteristics , 2003, Clinical Neurophysiology.

[8]  Carmen Vidaurre,et al.  BioSig: The Free and Open Source Software Library for Biomedical Signal Processing , 2011, Comput. Intell. Neurosci..

[9]  G. Pfurtscheller,et al.  Optimal spatial filtering of single trial EEG during imagined hand movement. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[10]  Geng Liu,et al.  Algorithm and Data Optimization Techniques for Scaling to Massively Threaded Systems , 2012, Computer.

[11]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[12]  Bryan Reimer,et al.  A Comparison of Heart Rate and Heart Rate Variability Indices in Distinguishing Single-Task Driving and Driving Under Secondary Cognitive Workload , 2017 .

[13]  R. Ward,et al.  EMG and EOG artifacts in brain computer interface systems: A survey , 2007, Clinical Neurophysiology.

[14]  Fabien Lotte,et al.  Brain-Computer Interfaces: Beyond Medical Applications , 2012, Computer.

[15]  Fabien Lotte,et al.  A Tutorial on EEG Signal Processing Techniques for Mental State Recognition in Brain-Computer Interfaces , 2014 .

[16]  Ivan Poupyrev,et al.  3D User Interfaces: Theory and Practice , 2004 .

[17]  Deborah Hix,et al.  Usability Evaluation in Virtual Environments: Classification and Comparison of Methods , 2001 .

[18]  Kathryn M. McMillan,et al.  N‐back working memory paradigm: A meta‐analysis of normative functional neuroimaging studies , 2005, Human brain mapping.

[19]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[20]  Cláudio T. Silva,et al.  A User Study of Visualization Effectiveness Using EEG and Cognitive Load , 2011, Comput. Graph. Forum.

[21]  Jean-Baptiste de la Rivière,et al.  CubTile: a multi-touch cubic interface , 2008, VRST '08.

[22]  Ryan O. Murphy,et al.  Using Skin Conductance in Judgment and Decision Making Research , 2011 .

[23]  Cuntai Guan,et al.  Regularizing Common Spatial Patterns to Improve BCI Designs: Unified Theory and New Algorithms , 2011, IEEE Transactions on Biomedical Engineering.

[24]  Wojciech Samek,et al.  Transferring Subspaces Between Subjects in Brain--Computer Interfacing , 2012, IEEE Transactions on Biomedical Engineering.

[25]  Laura Astolfi,et al.  Selected Papers from the 4th International Conference on Bioinspired Systems and Cognitive Signal Processing , 2011, Comput. Intell. Neurosci..

[26]  Martin Hachet,et al.  A Survey of Interaction Techniques for Interactive 3D Environments , 2013, Eurographics.

[27]  Robert J. K. Jacob,et al.  Using fNIRS brain sensing to evaluate information visualization interfaces , 2013, CHI.

[28]  Yang Wang,et al.  GSR and Blink Features for Cognitive Load Classification , 2013, INTERACT.

[29]  Roy Ruddle.  Review: 3D User Interfaces: Theory and Practice Doug A. Bowman , Ernst Kruijff , Joseph J. LaViola Jr. , Ivan Poupyrev , 2005 .

[30]  Ming-Syan Chen,et al.  ConvenienceProbe: A Phone-Based System for Retail Trade-Area Analysis , 2014, IEEE Pervasive Computing.

[31]  Ana L. N. Fred,et al.  Biosignals for Everyone , 2014, IEEE Pervasive Computing.

[32]  Guillaume Gibert,et al.  OpenViBE: An Open-Source Software Platform to Design, Test, and Use BrainComputer Interfaces in Real and Virtual Environments , 2010, PRESENCE: Teleoperators and Virtual Environments.

[33]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.