EEG-based Evaluation of Cognitive Workload Induced by Acoustic Parameters for Data Sonification

Data Visualization has been receiving growing attention recently, with ubiquitous smart devices designed to render information in a variety of ways. However, while evaluations of visual tools for their interpretability and intuitiveness have been commonplace, not much research has been devoted to other forms of data rendering, \eg, sonification. This work is the first to automatically estimate the cognitive load induced by different acoustic parameters considered for sonification in prior studies~\citeferguson2017evaluation,ferguson2018investigating. We examine cognitive load via (a) perceptual data-sound mapping accuracies of users for the different acoustic parameters, (b) cognitive workload impressions explicitly reported by users, and (c) their implicit EEG responses compiled during the mapping task. Our main findings are that (i) low cognitive load-inducing (ıe, more intuitive) acoustic parameters correspond to higher mapping accuracies, (ii) EEG spectral power analysis reveals higher α band power for low cognitive load parameters, implying a congruent relationship between explicit and implicit user responses, and (iii) Cognitive load classification with EEG features achieves a peak F1-score of 0.64, confirming that reliable workload estimation is achievable with user EEG data compiled using wearable sensors.

[1]  Subramanian Ramanathan,et al.  Gender and emotion recognition with implicit user signals , 2017, ICMI.

[2]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[3]  Stephen A. Brewster,et al.  Investigating Perceptual Congruence between Data and Display Dimensions in Sonification , 2018, CHI.

[4]  Subramanian Ramanathan,et al.  Watch to Edit: Video Retargeting using Gaze , 2018, Comput. Graph. Forum.

[5]  Stephen A. Brewster,et al.  Evaluation of psychoacoustic sound parameters for sonification , 2017, ICMI.

[6]  Mohammed Yeasin,et al.  Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks , 2015, ICLR.

[7]  Bruce N. Walker,et al.  Mappings and metaphors in auditory displays: An experimental assessment , 2005, TAP.

[8]  Gary S. Kendall Visualization by Ear: Auditory Imagery for Scientific Visualization and Virtual Reality , 1991 .

[9]  Michel Hersen,et al.  Introduction to Comprehensive Clinical Psychology , 1998 .

[10]  Jeha Ryu,et al.  Tactile Visualization with Mobile AR on a Handheld Device , 2007, HAID.

[11]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[12]  Jacek Gwizdka,et al.  Using Wireless EEG Signals to Assess Memory Workload in the $n$-Back Task , 2016, IEEE Transactions on Human-Machine Systems.

[13]  Weidong Huang,et al.  Using eye tracking to investigate graph layout effects , 2007, 2007 6th International Asia-Pacific Symposium on Visualization.

[14]  Robert J. K. Jacob,et al.  Using fNIRS brain sensing to evaluate information visualization interfaces , 2013, CHI.

[15]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[16]  Jeffrey Heer,et al.  Protovis: A Graphical Toolkit for Visualization , 2009, IEEE Transactions on Visualization and Computer Graphics.

[17]  Chi Thanh Vi,et al.  Error related negativity in observing interactive tasks , 2014, CHI.

[18]  Subramanian Ramanathan,et al.  An EEG-based Image Annotation System , 2017, NCVPRIPG.

[19]  Silke Paulmann,et al.  The Neurocognition of Prosody , 2015 .

[20]  Thomas Ertl,et al.  Visual analysis of perceptual and cognitive processes , 2014, 2014 International Conference on Information Visualization Theory and Applications (IVAPP).

[21]  Yang Wang,et al.  Robust Multimodal Cognitive Load Measurement , 2016, Human–Computer Interaction Series.

[22]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.

[23]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[24]  Cesare Furlanello,et al.  Deep learning for automatic stereotypical motor movement detection using wearable sensors in autism spectrum disorders , 2017, Signal Process..

[25]  M. Crocker,et al.  On the Proper Treatment of the N400 and P600 in Language Comprehension , 2017, Front. Psychol..

[26]  Cláudio T. Silva,et al.  A User Study of Visualization Effectiveness Using EEG and Cognitive Load , 2011, Comput. Graph. Forum.

[27]  Adrian David Cheok,et al.  Digital taste and smell communication , 2011, BODYNETS.