EEG-based Measures of Auditory Saliency in a Complex Context

Auditory saliency is an important mechanism that helps humans extract relevant information from environments. Audio notifications of mobile devices with high saliency can increase users' receptivity, yet overly high saliency could cause annoyance. Accurately measuring auditory saliency of a notification is critical for evaluating its usability. Previous studies adopted behavioral methods. However, their results may not accurately reflect auditory saliency as humans' perception of auditory saliency often involves complicated cognitive processes. Thus, we propose an electroencephalography (EEG)-based approach that can complement behavioral studies to provide a more nuanced analysis of auditory saliency. We evaluated our method by conducting an EEG experiment that measured the mismatch negativity and P3a of the sounds in realistic scenarios. We also conducted a behavioral experiment to link the EEG-based method with the behavioral method. The results suggested that EEG can provide detailed information about how human perceive auditory saliency and complement the behavioral measures.

[1]  W. Singer,et al.  Dynamic predictions: Oscillations and synchrony in top–down processing , 2001, Nature Reviews Neuroscience.

[2]  William W. Gaver.  How Do We Hear in the World?: Explorations in Ecological Acoustics , 1993 .

[3]  I. Winkler,et al.  Involuntary Attention and Distractibility as Evaluated with Event-Related Brain Potentials , 2000, Audiology and Neurotology.

[4]  Li-Wei Ko,et al.  An EEG-based approach for evaluating audio notifications under ambient sounds , 2014, CHI.

[5]  H Lyytinen,et al.  Is the appearance of mismatch negativity during stage 2 sleep related to the elicitation of K-complex? , 1994, Electroencephalography and clinical neurophysiology.

[6]  S. Shamma,et al.  Interaction between Attention and Bottom-Up Saliency Mediates the Representation of Foreground and Background in an Auditory Scene , 2009, PLoS biology.

[7]  Mary Czerwinski,et al.  Instant Messaging and Interruption: Influence of Task Type on Performance , 2000 .

[8]  Anna M. Bianchi,et al.  ICA-derived cortical responses indexing rapid multi-feature auditory processing in six-month-old infants , 2016, NeuroImage.

[9]  C. C. Duncan,et al.  Event-related potentials in clinical research: Guidelines for eliciting, recording, and quantifying mismatch negativity, P300, and N400 , 2009, Clinical Neurophysiology.

[10]  Anthony J. Rissling,et al.  Cortical substrates and functional correlates of auditory deviance processing deficits in schizophrenia , 2014, NeuroImage: Clinical.

[11]  J. Joutsensalo,et al.  Optimal Digital Filtering versus Difference Waves on the Mismatch Negativity in an Uninterrupted Sound Paradigm , 2007, Developmental neuropsychology.

[12]  R. Näätänen,et al.  The mismatch negativity (MMN) in basic research of central auditory processing: A review , 2007, Clinical Neurophysiology.

[13]  Michael T. Lippert,et al.  Mechanisms for Allocating Auditory Attention: An Auditory Saliency Map , 2005, Current Biology.

[14]  Jung-Tai King,et al.  Measuring the Influences of Musical Parameters on Cognitive and Behavioral Responses to Audio Notifications Using EEG and Large-scale Online Studies , 2019, CHI.

[15]  John C. Tang,et al.  Investigating Mobile Users' Ringer Mode Usage and Attentiveness and Responsiveness to Communication , 2015, MobileHCI.

[16]  R. Näätänen,et al.  Short-term habituation and dishabituation of the mismatch negativity of the ERP. , 1984, Psychophysiology.

[17]  David V. Anderson,et al.  Identifying salient sounds using dual-task experiments , 2013, 2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.

[18]  Mirco Musolesi,et al.  My Phone and Me: Understanding People's Receptivity to Mobile Notifications , 2016, CHI.

[19]  Gabriel Curio,et al.  Does familiarity facilitate the cortical processing of music sounds? , 2004, Neuroreport.

[20]  Bert De Coensel,et al.  A model of saliency-based auditory attention to environmental sound , 2010 .

[21]  Kenneth Kreutz-Delgado,et al.  EyeCatch: Data-mining over half a million EEG independent components to construct a fully-automated eye-component detector , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[22]  M. Tervaniemi,et al.  The sound of music: Differentiating musicians using a fast, musical multi-feature mismatch negativity paradigm , 2012, Neuropsychologia.

[23]  Mary Czerwinski,et al.  Notification, Disruption, and Memory: Effects of Messaging Interruptions on Memory and Performance , 2001, INTERACT.

[24]  Tom Eichele,et al.  Semi-automatic identification of independent components representing EEG artifact , 2009, Clinical Neurophysiology.

[25]  Jörg Kliewer,et al.  Assessing subjective perception of audio quality by measuring the information flow on the brain-response channel , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[26]  Shrikanth S. Narayanan,et al.  A saliency-based auditory attention model with applications to unsupervised prominent syllable detection in speech , 2007, INTERSPEECH.

[27]  S. R. Butler,et al.  Electrophysiological indicator of awakening from coma , 1993, The Lancet.

[28]  Thomas Sandholm,et al.  Toward An Improved Model Of Auditory Saliency , 2013 .

[29]  Varinthira Duangudom Delmotte.  Computational auditory saliency , 2012 .

[30]  Jung-Tai King,et al.  To Repeat or Not to Repeat?: Redesigning Repeating Auditory Alarms Based on EEG Analysis , 2019, CHI.

[31]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[32]  V. Sinha,et al.  Event-related potential: An overview , 2009, Industrial psychiatry journal.

[33]  Emine Merve Kaya,et al.  Investigating bottom-up auditory attention , 2014, Front. Hum. Neurosci..

[34]  David V. Anderson,et al.  Using auditory saliency to understand complex auditory scenes , 2007, 2007 15th European Signal Processing Conference.

[35]  Eckart Altenmüller,et al.  Changes in emotional tone and instrumental timbre are reflected by the mismatch negativity. , 2004, Brain research. Cognitive brain research.

[36]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[37]  J. Polich.  Updating P300: An integrative theory of P3a and P3b , 2007, Clinical Neurophysiology.

[38]  M. Tervaniemi,et al.  The mismatch negativity in cognitive and clinical neuroscience: Theoretical and methodological considerations , 2007, Biological Psychology.

[39]  Terrence J. Sejnowski,et al.  Independent Component Analysis of Simulated ERP Data , 2000 .

[40]  Mounya Elhilali,et al.  Modelling auditory attention , 2017, Philosophical Transactions of the Royal Society B: Biological Sciences.

[41]  R. Näätänen,et al.  Measurement of extensive auditory discrimination profiles using the mismatch negativity (MMN) of the auditory event-related potential (ERP) , 2007, Clinical Neurophysiology.

[42]  Fahim Kawsar,et al.  The myth of subtle notifications , 2014, UbiComp Adjunct.