Linear Modeling of Neurophysiological Responses to Speech and Other Continuous Stimuli: Methodological Considerations for Applied Research

Cognitive neuroscience, in particular research on speech and language, has seen an increase in the use of linear modeling techniques for studying the processing of natural, environmental stimuli. The availability of such computational tools has prompted similar investigations in many clinical domains, facilitating the study of cognitive and sensory deficits under more naturalistic conditions. However, studying clinical (and often highly heterogeneous) cohorts introduces an added layer of complexity to such modeling procedures, potentially leading to instability of such techniques and, as a result, inconsistent findings. Here, we outline some key methodological considerations for applied research, referring to a hypothetical clinical experiment involving speech processing and worked examples of simulated electrophysiological (EEG) data. In particular, we focus on experimental design, data preprocessing, stimulus feature extraction, model design, model training and evaluation, and interpretation of model weights. Throughout the paper, we demonstrate the implementation of each step in MATLAB using the mTRF-Toolbox and discuss how to address issues that could arise in applied research. In doing so, we hope to provide better intuition on these more technical points and provide a resource for applied and clinical researchers investigating sensory and cognitive processing using ecologically rich stimuli.

[1]  Anna A. Ivanova,et al.  Beyond linear regression: mapping models in cognitive neuroscience should align with research goals , 2021, bioRxiv.

[2]  Michael J. Crosse,et al.  Neurophysiological Indices of Audiovisual Speech Processing Reveal a Hierarchy of Multisensory Integration Effects , 2021, The Journal of Neuroscience.

[3]  Liberty S. Hamilton,et al.  Generalizable EEG Encoding Models with Naturalistic Audiovisual Stimuli , 2021, The Journal of Neuroscience.

[4]  K. Harris Nonsense correlations in neuroscience , 2020, bioRxiv.

[5]  Jeffrey Mark Siskind,et al.  The Perils and Pitfalls of Block Design for EEG Classification Experiments , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  S. Anderson,et al.  Objective evidence of temporal processing deficits in older adults , 2020, Hearing Research.

[7]  Edmund C. Lalor,et al.  Dissociable electrophysiological measures of natural language processing reveal differences in speech comprehension strategy in healthy ageing , 2020, Scientific Reports.

[8]  Don A. Harrington,et al.  Hearing Impairment , 2020, Pediatric Practice Guidelines.

[9]  A. Cheveigné ZapLine: A simple and effective method to remove power line artifacts , 2020, NeuroImage.

[10]  Jonathan Z. Simon,et al.  Neural speech restoration at the cocktail party: Auditory cortex recovers masked speech of both attended and ignored speakers , 2019, bioRxiv.

[11]  Alexander Bertrand,et al.  Effect of number and placement of EEG electrodes on measurement of neural tracking of speech , 2019, bioRxiv.

[12]  C. Kayser,et al.  Neural Entrainment and Attentional Selection in the Listening Brain , 2019, Trends in Cognitive Sciences.

[13]  Surya Ganguli,et al.  A deep learning framework for neuroscience , 2019, Nature Neuroscience.

[14]  Alain de Cheveigné,et al.  ZapLine: A simple and effective method to remove power line artifacts , 2019, NeuroImage.

[15]  N. Mesgarani,et al.  Cortical encoding of melodic expectations in human temporal cortex , 2019, bioRxiv.

[16]  Tom Francart,et al.  Evidence for enhanced neural tracking of the speech envelope underlying age-related speech-in-noise difficulties. , 2019, Journal of neurophysiology.

[17]  Alain de Cheveigné,et al.  Filters: When, Why, and How (Not) to Use Them , 2019, Neuron.

[18]  Lucas C. Parra,et al.  Multiway canonical correlation analysis of brain data , 2019, NeuroImage.

[19]  Tom Francart,et al.  Hearing impairment is associated with enhanced neural tracking of the speech envelope , 2019, Hearing Research.

[20]  R. Tibshirani,et al.  (37) Medications as Independent Risk Factors of Delirium in Patients With COVID-19: A Retrospective Study , 2018, Journal of the Academy of Consultation-Liaison Psychiatry.

[21]  Nikolaus Kriegeskorte,et al.  Interpreting encoding and decoding models , 2018, Current Opinion in Neurobiology.

[22]  L. Elliot Hong,et al.  Rapid Transformation from Auditory to Linguistic Representations of Continuous Speech , 2018, Current Biology.

[23]  Alexander Bertrand,et al.  Utility Metrics for Assessment and Subset Selection of Input Variables for Linear Estimation [Tips & Tricks] , 2018, IEEE Signal Processing Magazine.

[24]  Edmund C. Lalor,et al.  Neural tracking of auditory motion is reflected by delta phase and alpha power of EEG , 2018, NeuroImage.

[25]  Jack L. Gallant,et al.  Voxelwise encoding models with non-spherical multivariate normal priors , 2018, NeuroImage.

[26]  Daniel D. E. Wong,et al.  A Comparison of Regularization Methods in Forward and Backward Models for Auditory Attention Decoding , 2018, Front. Neurosci..

[27]  Liberty S. Hamilton,et al.  The revolution will not be controlled: natural stimuli in speech neuroscience , 2018, Language, cognition and neuroscience.

[28]  Denis Burnham,et al.  Atypical cortical entrainment to speech in the right hemisphere underpins phonemic deficits in dyslexia , 2018, NeuroImage.

[29]  Ben Somers,et al.  Neural tracking of the speech envelope in cochlear implant users , 2018, bioRxiv.

[30]  Alain de Cheveigné,et al.  Decoding the auditory brain with canonical component analysis , 2017, NeuroImage.

[31]  Satrajit S. Ghosh,et al.  FMRIPrep: a robust preprocessing pipeline for functional MRI , 2018, Nature Methods.

[32]  Edmund C Lalor,et al.  Cortical Measures of Phoneme-Level Speech Encoding Correlate with the Perceived Clarity of Natural Speech , 2018, eNeuro.

[33]  Robert Oostenveld,et al.  Integrated analysis of anatomical and electrophysiological human intracranial data , 2017, Nature Protocols.

[34]  Jonathan Z. Simon,et al.  Real-Time Tracking of Selective Auditory Attention From M/EEG: A Bayesian Filtering Approach , 2017, bioRxiv.

[35]  Robert T. Knight,et al.  Encoding and Decoding Models in Cognitive Electrophysiology , 2017, Front. Syst. Neurosci..

[36]  Edmund C. Lalor,et al.  Electrophysiological Correlates of Semantic Dissimilarity Reflect the Comprehension of Natural, Narrative Speech , 2017, Current Biology.

[37]  Adrian K. C. Lee,et al.  Auditory Brainstem Responses to Continuous Natural Speech in Human Listeners , 2017, eNeuro.

[38]  Edmund C. Lalor,et al.  Indexing cortical entrainment to natural speech at the phonemic level: Methodological considerations for applied research , 2017, Hearing Research.

[39]  Alexander Bertrand,et al.  Auditory-Inspired Speech Envelope Extraction Methods for Improved EEG-Based Auditory Attention Detection in a Cocktail Party Scenario , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[40]  Edmund C. Lalor,et al.  Visual Cortical Entrainment to Motion and Categorical Speech Features during Silent Lipreading , 2017, Front. Hum. Neurosci..

[41]  Edmund C. Lalor,et al.  The Multivariate Temporal Response Function (mTRF) Toolbox: A MATLAB Toolbox for Relating Neural Signals to Continuous Stimuli , 2016, Front. Hum. Neurosci..

[42]  Jack L. Gallant,et al.  Decoding the Semantic Content of Natural Movies from Human Brain Activity , 2016, Frontiers in systems neuroscience.

[43]  Michael J. Crosse,et al.  Eye Can Hear Clearly Now: Inverse Effectiveness in Natural Audiovisual Speech Processing Relies on Long-Term Crossmodal Temporal Integration , 2016, The Journal of Neuroscience.

[44]  J. DiCarlo,et al.  Using goal-driven deep learning models to understand sensory cortex , 2016, Nature Neuroscience.

[45]  Klaus-Robert Müller,et al.  On the influence of high-pass filtering on ICA-based artifact reduction in EEG-ERP , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[46]  Robin A A Ince,et al.  Irregular Speech Rate Dissociates Auditory Cortical Entrainment, Evoked Responses, and Frontal Alpha , 2015, The Journal of Neuroscience.

[47]  Daniel Povey,et al.  MUSAN: A Music, Speech, and Noise Corpus , 2015, ArXiv.

[48]  Michael J. Crosse,et al.  Congruent Visual Speech Enhances Cortical Entrainment to Continuous Auditory Speech in Noise-Free Conditions , 2015, The Journal of Neuroscience.

[49]  Edmund C. Lalor,et al.  Low-Frequency Cortical Entrainment to Speech Reflects Phoneme-Level Processing , 2015, Current Biology.

[50]  Karim Jerbi,et al.  Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy , 2015, Journal of Neuroscience Methods.

[51]  John J. Foxe,et al.  Attentional Selection in a Cocktail Party Environment Can Be Decoded from Single-Trial EEG. , 2015, Cerebral cortex.

[52]  Kyungmin Su,et al.  The PREP pipeline: standardized preprocessing for large-scale EEG analysis , 2015, Front. Neuroinform..

[53]  James A. O'Sullivan,et al.  Evidence for Neural Computations of Temporal Coherence in an Auditory Scene and Their Enhancement during Active Listening , 2015, The Journal of Neuroscience.

[54]  Naomi Harte,et al.  TCD-TIMIT: An Audio-Visual Corpus of Continuous Speech , 2015, IEEE Transactions on Multimedia.

[55]  Lucas C. Parra,et al.  Joint decorrelation, a versatile tool for multichannel data analysis , 2014, NeuroImage.

[56]  John J. Foxe,et al.  Towards obtaining spatiotemporally precise responses to continuous sensory stimuli in humans: A general linear modeling approach to EEG , 2014, NeuroImage.

[57]  J. Simon,et al.  Cortical entrainment to continuous speech: functional roles and interpretations , 2014, Front. Hum. Neurosci..

[58]  Jonathan Z. Simon,et al.  Robust cortical entrainment to the speech envelope relies on the spectro-temporal fine structure , 2014, NeuroImage.

[59]  Stefan Haufe,et al.  On the interpretation of weight vectors of linear models in multivariate neuroimaging , 2014, NeuroImage.

[60]  Edmund C. Lalor,et al.  The effects of attention and visual input on the representation of natural speech in EEG , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[61]  John J. Foxe,et al.  Atypical cortical representation of peripheral visual space in children with an autism spectrum disorder , 2013, The European journal of neuroscience.

[62]  Jonathan Z. Simon,et al.  Adaptive Temporal Encoding Leads to a Background-Insensitive Cortical Representation of Speech , 2013, The Journal of Neuroscience.

[63]  David Poeppel,et al.  Cortical oscillations and speech processing: emerging computational principles and operations , 2012, Nature Neuroscience.

[64]  Ross Gibson,et al.  What the eye can hear , 2011 .

[65]  John J. Foxe,et al.  Neural responses to uninterrupted natural speech can be extracted with precise temporal resolution , 2010, The European journal of neuroscience.

[66]  S. David,et al.  Influence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex. , 2009, Journal of neurophysiology.

[67]  John J. Foxe,et al.  Resolving precise temporal processing properties of the auditory system using continuous stimuli. , 2009, Journal of neurophysiology.

[68]  Denis Brunet,et al.  Topographic ERP Analyses: A Step-by-Step Tutorial Review , 2008, Brain Topography.

[69]  Nima Mesgarani,et al.  Phoneme representation and classification in primary auditory cortex. , 2008, The Journal of the Acoustical Society of America.

[70]  Barak A. Pearlmutter,et al.  Dissecting the cellular contributions to early visual sensory processing deficits in schizophrenia using the VESPA evoked response , 2008, Schizophrenia Research.

[71]  R. Oostenveld,et al.  Nonparametric statistical testing of EEG- and MEG-data , 2007, Journal of Neuroscience Methods.

[72]  D. Poeppel,et al.  The cortical organization of speech processing , 2007, Nature Reviews Neuroscience.

[73]  S. David,et al.  Estimating sparse spectro-temporal receptive fields with natural stimuli , 2007, Network.

[74]  Roy D. Patterson,et al.  A Dynamic Compressive Gammachirp Auditory Filterbank , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[75]  Barak A. Pearlmutter,et al.  The VESPA: A method for the rapid estimation of a visual evoked potential , 2006, NeuroImage.

[76]  J. Gallant,et al.  Complete functional characterization of sensory neurons by system identification. , 2006, Annual review of neuroscience.

[77]  Nicole C. Rust,et al.  Do We Know What the Early Visual System Does? , 2005, The Journal of Neuroscience.

[78]  W. Bialek,et al.  Features and dimensions: Motion estimation in fly vision , 2005, q-bio/0505003.

[79]  Christian K. Machens,et al.  Linearity of Cortical Receptive Fields Measured with Natural Sounds , 2004, The Journal of Neuroscience.

[80]  David Poeppel,et al.  The analysis of speech in different temporal integration windows: cerebral lateralization as 'asymmetric sampling in time' , 2003, Speech Commun..

[81]  Thomas E. Nichols,et al.  Nonparametric permutation tests for functional neuroimaging: A primer with examples , 2002, Human brain mapping.

[82]  N. C. Singh,et al.  Estimating spatio-temporal receptive fields of auditory and visual neurons from their responses to natural stimuli , 2001 .

[83]  Erkki Oja,et al.  Independent component analysis: algorithms and applications , 2000, Neural Networks.

[84]  R Drullman,et al.  Temporal envelope and fine structure cues for speech intelligibility. , 1994, The Journal of the Acoustical Society of America.

[85]  R. Ilmoniemi,et al.  Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain , 1993 .

[86]  I. Hashimoto [Auditory brainstem responses]. , 1985, No to shinkei = Brain and nerve.

[87]  S. S. Stevens The Measurement of Loudness , 1955 .

[88]  Edmund C. Lalor,et al.  Linear-nonlinear Bernoulli modeling for quantifying temporal coding of phonemes in brain responses to continuous speech , 2019, 2019 Conference on Cognitive Computational Neuroscience.

[89]  J. Simon,et al.  Neural coding of continuous speech in auditory cortex during monaural and dichotic listening. , 2012, Journal of neurophysiology.

[90]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.