EEG can predict speech intelligibility

OBJECTIVE Speech signals have a remarkable ability to entrain brain activity to the rapid fluctuations of speech sounds. For instance, one can readily measure a correlation of the sound amplitude with the evoked responses of the electroencephalogram (EEG), and the strength of this correlation is indicative of whether the listener is attending to the speech. In this study we asked whether this stimulus-response correlation is also predictive of speech intelligibility. APPROACH We hypothesized that when a listener fails to understand the speech in adverse hearing conditions, attention wanes and stimulus-response correlation also drops. To test this, we measure a listener's ability to detect words in noisy speech while recording their brain activity using EEG. We alter intelligibility without changing the acoustic stimulus by pairing it with congruent and incongruent visual speech. MAIN RESULTS For almost all subjects we found that an improvement in speech detection coincided with an increase in correlation between the noisy speech and the EEG measured over a period of 30 min. SIGNIFICANCE We conclude that simultaneous recordings of the perceived sound and the corresponding EEG response may be a practical tool to assess speech intelligibility in the context of hearing aids.

[1]  Birger Kollmeier,et al.  Machine learning for decoding listeners’ attention from electroencephalography evoked by continuous speech , 2020, The European journal of neuroscience.

[2]  Preben Kidmose,et al.  Auditory evoked responses from Ear-EEG recordings , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[3]  H. Hotelling.  Relations Between Two Sets of Variates , 1936 .

[4]  John J. Foxe,et al.  Neural responses to uninterrupted natural speech can be extracted with precise temporal resolution , 2010, The European journal of neuroscience.

[5]  Jeffery A. Jones,et al.  Visual Prosody and Speech Intelligibility , 2004, Psychological science.

[6]  Joachim Gross,et al.  Phase-Locked Responses to Speech in Human Auditory Cortex are Enhanced During Comprehension , 2012, Cerebral cortex.

[7]  Stefan L. Frank,et al.  Lexical representation explains cortical entrainment during speech comprehension , 2017, PloS one.

[8]  Alexander Bertrand,et al.  EEG-Informed Attended Speaker Extraction From Recorded Speech Mixtures With Application in Neuro-Steered Hearing Prostheses , 2016, IEEE Transactions on Biomedical Engineering.

[9]  G. Pazour,et al.  Ror2 signaling regulates Golgi structure and transport through IFT20 for tumor invasiveness , 2017, Scientific Reports.

[10]  Matthew H. Davis,et al.  Phase Entrainment of Brain Oscillations Causally Modulates Neural Responses to Intelligible Speech , 2018, Current Biology.

[11]  David Poeppel,et al.  Acoustic landmarks drive delta–theta oscillations to enable speech comprehension by facilitating perceptual parsing , 2014, NeuroImage.

[12]  John J. Foxe,et al.  At what time is the cocktail party? A late locus of selective attention to natural speech , 2012, The European journal of neuroscience.

[13]  Tobias Reichenbach,et al.  The human auditory brainstem response to running speech reveals a subcortical mechanism for selective attention , 2017, bioRxiv.

[14]  Jonathan Z. Simon,et al.  Adaptive Temporal Encoding Leads to a Background-Insensitive Cortical Representation of Speech , 2013, The Journal of Neuroscience.

[15]  Zhong-Qiu Wang,et al.  Multi-Channel Deep Clustering: Discriminative Spectral and Spatial Embeddings for Speaker-Independent Speech Separation , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[16]  D. Poeppel,et al.  Phase Patterns of Neuronal Responses Reliably Discriminate Speech in Human Auditory Cortex , 2007, Neuron.

[17]  N. Mesgarani,et al.  Selective cortical representation of attended speaker in multi-talker speech perception , 2012, Nature.

[18]  John J. Foxe,et al.  Attentional Selection in a Cocktail Party Environment Can Be Decoded from Single-Trial EEG. , 2015, Cerebral cortex.

[19]  Lucas C. Parra,et al.  EEG in the classroom: Synchronised neural recordings during video presentation , 2016, Scientific Reports.

[20]  Alexandre Hyafil,et al.  Speech encoding by coupled cortical theta and gamma oscillations , 2015, eLife.

[21]  Sharon M. Thomas,et al.  Contributions of oral and extraoral facial movement to visual and audiovisual speech perception. , 2004, Journal of experimental psychology. Human perception and performance.

[22]  Denis G. Pelli,et al.  ECVP '07 Abstracts , 2007, Perception.

[23]  Nicola Molinaro,et al.  Delta(but not theta)‐band cortical entrainment involves speech‐specific processing , 2018, The European journal of neuroscience.

[24]  Luc H. Arnal,et al.  Cortical oscillations and sensory predictions , 2012, Trends in Cognitive Sciences.

[25]  L. Auger.  The Journal of the Acoustical Society of America , 1949 .

[26]  John J. Foxe,et al.  Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. , 2006, Cerebral cortex.

[27]  M. D'Zmura,et al.  Envelope responses in single-trial EEG indicate attended speaker in a ‘cocktail party’ , 2014, Journal of Neural Engineering.

[28]  Oded Ghitza,et al.  On the Role of Theta-Driven Syllabic Parsing in Decoding Speech: Intelligibility of Speech with a Manipulated Modulation Spectrum , 2012, Front. Psychology.

[29]  C. Schroeder,et al.  Attention modulates ‘speech-tracking’ at a cocktail party , 2012, Trends in Cognitive Sciences.

[30]  J. Simon,et al.  Cortical entrainment to continuous speech: functional roles and interpretations , 2014, Front. Hum. Neurosci..

[31]  Preben Kidmose,et al.  Ear-EEG: User-Centered and Wearable BCI , 2014 .

[32]  Lucas C. Parra,et al.  Extracting multidimensional stimulus-response correlations using hybrid encoding-decoding of neural activity , 2017, NeuroImage.

[33]  P. Hagoort,et al.  Neural entrainment determines the words we hear , 2017, bioRxiv.

[34]  Gregor Thut,et al.  Lip movements entrain the observers’ low-frequency brain oscillations to facilitate speech intelligibility , 2016, eLife.

[35]  Michael J. Crosse,et al.  Eye Can Hear Clearly Now: Inverse Effectiveness in Natural Audiovisual Speech Processing Relies on Long-Term Crossmodal Temporal Integration , 2016, The Journal of Neuroscience.

[36]  D. Poeppel,et al.  Mechanisms Underlying Selective Neuronal Tracking of Attended Speech at a “Cocktail Party” , 2013, Neuron.

[37]  Jason J. Ki,et al.  Attention Strongly Modulates Reliability of Neural Responses to Naturalistic Narrative Stimuli , 2016, The Journal of Neuroscience.

[38]  S. Haegens,et al.  Rhythmic facilitation of sensory processing: A critical review , 2017, Neuroscience & Biobehavioral Reviews.

[39]  Ying-Yee Kong,et al.  Effects of Spectral Degradation on Attentional Modulation of Cortical Auditory Responses to Continuous Speech , 2015, Journal of the Association for Research in Otolaryngology.

[40]  S. Debener,et al.  How about taking a low-cost, small, and wireless EEG for a walk? , 2012, Psychophysiology.

[41]  D. Poeppel,et al.  Cortical Tracking of Hierarchical Linguistic Structures in Connected Speech , 2015, Nature Neuroscience.

[42]  Michael J. Crosse,et al.  Congruent Visual Speech Enhances Cortical Entrainment to Continuous Auditory Speech in Noise-Free Conditions , 2015, The Journal of Neuroscience.

[43]  John J. Foxe,et al.  Visual evoked spread spectrum analysis (VESPA) responses to stimuli biased towards magnocellular and parvocellular pathways , 2009, Vision Research.

[44]  L. Parra,et al.  Engaging narratives evoke similar neural activity and lead to similar time perception , 2017, bioRxiv.

[45]  Stefan Debener,et al.  Target Speaker Detection with Concealed EEG Around the Ear , 2016, Front. Neurosci..

[46]  Lei Ai,et al.  Age and sex modulate the variability of neural responses to naturalistic videos , 2017, bioRxiv.

[47]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[48]  Christopher K. Kovach,et al.  Temporal Envelope of Time-Compressed Speech Represented in the Human Auditory Cortex , 2009, The Journal of Neuroscience.

[49]  J. Simon,et al.  Neural coding of continuous speech in auditory cortex during monaural and dichotic listening. , 2012, Journal of neurophysiology.

[50]  Preben Kidmose,et al.  In-Ear EEG From Viscoelastic Generic Earpieces: Robust and Unobtrusive 24/7 Monitoring , 2016, IEEE Sensors Journal.

[51]  Lei Ai,et al.  The Variability of Neural Responses to Naturalistic Videos Change with Age and Sex , 2018, eNeuro.

[52]  Alexander Bertrand,et al.  Auditory-Inspired Speech Envelope Extraction Methods for Improved EEG-Based Auditory Attention Detection in a Cocktail Party Scenario , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[53]  Jonathan Z. Simon,et al.  Robust cortical entrainment to the speech envelope relies on the spectro-temporal fine structure , 2014, NeuroImage.

[54]  Jan Wouters,et al.  Speech Intelligibility Predicted from Neural Entrainment of the Speech Envelope , 2018, bioRxiv.

[55]  Oded Ghitza,et al.  Entrained theta oscillations guide perception of subsequent speech: behavioural evidence from rate normalisation , 2018 .

[56]  M. Wibral,et al.  Untangling cross-frequency coupling in neuroscience , 2014, Current Opinion in Neurobiology.

[57]  Zhuo Chen,et al.  Neural decoding of attentional selection in multi-speaker environments without access to separated sources , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[58]  Edmund C Lalor,et al.  Cortical Measures of Phoneme-Level Speech Encoding Correlate with the Perceived Clarity of Natural Speech , 2018, eNeuro.

[59]  R. Knight,et al.  The functional role of cross-frequency coupling , 2010, Trends in Cognitive Sciences.