The Multivariate Temporal Response Function (mTRF) Toolbox: A MATLAB Toolbox for Relating Neural Signals to Continuous Stimuli

Understanding how brains process sensory signals in natural environments is one of the key goals of twenty-first century neuroscience. While brain imaging and invasive electrophysiology will play key roles in this endeavor, there is also an important role to be played by noninvasive, macroscopic techniques with high temporal resolution such as electro- and magnetoencephalography. But challenges exist in determining how best to analyze such complex, time-varying neural responses to complex, time-varying and multivariate natural sensory stimuli. There has been a long history of applying system identification techniques to relate the firing activity of neurons to complex sensory stimuli and such techniques are now seeing increased application to EEG and MEG data. One particular example involves fitting a filter—often referred to as a temporal response function—that describes a mapping between some feature(s) of a sensory stimulus and the neural response. Here, we first briefly review the history of these system identification approaches and describe a specific technique for deriving temporal response functions known as regularized linear regression. We then introduce a new open-source toolbox for performing this analysis. We describe how it can be used to derive (multivariate) temporal response functions describing a mapping between stimulus and response in both directions. We also explain the importance of regularizing the analysis and how this regularization can be optimized for a particular dataset. We then outline specifically how the toolbox implements these analyses and provide several examples of the types of results that the toolbox can produce. Finally, we consider some of the limitations of the toolbox and opportunities for future development and application.

[1]  S. David,et al.  Influence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex. , 2009, Journal of neurophysiology.

[2]  John J. Foxe,et al.  Attentional Selection in a Cocktail Party Environment Can Be Decoded from Single-Trial EEG. , 2015, Cerebral cortex.

[3]  J. Simon,et al.  Emergence of neural encoding of auditory objects while listening to competing speakers , 2012, Proceedings of the National Academy of Sciences.

[4]  J. Gallant,et al.  Complete functional characterization of sensory neurons by system identification. , 2006, Annual review of neuroscience.

[5]  D. Poeppel,et al.  Phase Patterns of Neuronal Responses Reliably Discriminate Speech in Human Auditory Cortex , 2007, Neuron.

[6]  H. Barlow,et al.  Single Units and Sensation: A Neuron Doctrine for Perceptual Psychology? , 1972, Perception.

[7]  John J. Foxe,et al.  Resolving precise temporal processing properties of the auditory system using continuous stimuli. , 2009, Journal of neurophysiology.

[8]  William Bialek,et al.  Reading a Neural Code , 1991, NIPS.

[9]  J. Simon,et al.  Neural coding of continuous speech in auditory cortex during monaural and dichotic listening. , 2012, Journal of neurophysiology.

[10]  Lucas C. Parra,et al.  Joint decorrelation, a versatile tool for multichannel data analysis , 2014, NeuroImage.

[11]  J. Fermaglich Electric Fields of the Brain: The Neurophysics of EEG , 1982 .

[12]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[13]  R. Salmelin Clinical neurophysiology of language: The MEG approach , 2007, Clinical Neurophysiology.

[14]  G B Stanley,et al.  Reconstruction of Natural Scenes from Ensemble Responses in the Lateral Geniculate Nucleus , 1999, The Journal of Neuroscience.

[15]  J. Eggermont,et al.  Cross-correlation and joint spectro-temporal receptive field properties in auditory cortex. , 2005, Journal of neurophysiology.

[16]  Aroh Barjatya,et al.  Block Matching Algorithms For Motion Estimation , 2004 .

[17]  S. David,et al.  Estimating sparse spectro-temporal receptive fields with natural stimuli , 2007, Network.

[18]  D. Lehmann,et al.  Reference-free identification of components of checkerboard-evoked multichannel potential fields. , 1980, Electroencephalography and clinical neurophysiology.

[19]  Stefan Haufe,et al.  SPoC: A novel framework for relating the amplitude of neuronal oscillations to behaviorally relevant parameters , 2014, NeuroImage.

[20]  D. Poeppel,et al.  Mechanisms Underlying Selective Neuronal Tracking of Attended Speech at a “Cocktail Party” , 2013, Neuron.

[21]  Vasilis Z. Marmarelis,et al.  Nonlinear Dynamic Modeling of Physiological Systems: Marmarelis/Nonlinear , 2004 .

[22]  John J. Foxe,et al.  At what time is the cocktail party? A late locus of selective attention to natural speech , 2012, The European journal of neuroscience.

[23]  N. Logothetis,et al.  Natural vision reveals regional specialization to local motion and to contrast-invariant, global flow in the human brain. , 2008, Cerebral cortex.

[24]  Kuansan Wang,et al.  Auditory representations of acoustic signals , 1992, IEEE Trans. Inf. Theory.

[25]  Michael J. Crosse,et al.  Congruent Visual Speech Enhances Cortical Entrainment to Continuous Auditory Speech in Noise-Free Conditions , 2015, The Journal of Neuroscience.

[26]  Maarten De Vos,et al.  Decoding the attended speech stream with multi-channel EEG: implications for online, daily-life applications , 2015, Journal of neural engineering.

[27]  Jonathan Z. Simon,et al.  Robust cortical entrainment to the speech envelope relies on the spectro-temporal fine structure , 2014, NeuroImage.

[28]  Dario L. Ringach,et al.  Reverse correlation in neurophysiology , 2004, Cogn. Sci..

[29]  Lucas C. Parra,et al.  Recipes for the linear analysis of EEG , 2005, NeuroImage.

[30]  J. J. Eggermont,et al.  Quantitative characterisation procedure for auditory neurons based on the spectro-temporal receptive field , 1983, Hearing Research.

[31]  S A Shamma,et al.  Spectro-temporal response field characterization with dynamic ripples in ferret primary auditory cortex. , 2001, Journal of neurophysiology.

[32]  Edmund C. Lalor,et al.  Comparing linear and quadratic models of the human auditory system using EEG , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[33]  Brian N. Pasley,et al.  Decoding spectrotemporal features of overt and covert speech from the human cortex , 2014, Front. Neuroeng..

[34]  Christian K. Machens,et al.  Linearity of Cortical Receptive Fields Measured with Natural Sounds , 2004, The Journal of Neuroscience.

[35]  J. Gallant,et al.  Predicting neuronal responses during natural vision , 2005, Network.

[36]  W. Freeman,et al.  Spatial spectra of scalp EEG and EMG from awake humans , 2003, Clinical Neurophysiology.

[37]  N. Mesgarani,et al.  Selective cortical representation of attended speaker in multi-talker speech perception , 2012, Nature.

[38]  John J. Foxe,et al.  Atypical cortical representation of peripheral visual space in children with an autism spectrum disorder , 2013, The European journal of neuroscience.

[39]  J. P. Jones,et al.  The two-dimensional spatial structure of simple receptive fields in cat striate cortex. , 1987, Journal of neurophysiology.

[40]  John J. Foxe,et al.  Visual sensory processing deficits in schizophrenia: Is there anything to the magnocellular account? , 2012, Schizophrenia Research.

[41]  W. Bialek,et al.  Naturalistic stimuli increase the rate and efficiency of information transmission by primary auditory afferents , 1995, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[42]  P. Z. Marmarelis,et al.  Analysis of Physiological Systems: The White-Noise Approach , 2011 .

[43]  Stefan Haufe,et al.  On the interpretation of weight vectors of linear models in multivariate neuroimaging , 2014, NeuroImage.

[44]  Marina Schmid,et al.  An Introduction To The Event Related Potential Technique , 2016 .

[45]  John J. Foxe,et al.  Towards obtaining spatiotemporally precise responses to continuous sensory stimuli in humans: A general linear modeling approach to EEG , 2014, NeuroImage.

[46]  Guillermo Sapiro,et al.  A subspace reverse-correlation technique for the study of visual neurons , 1997, Vision Research.

[47]  E Ahissar,et al.  Speech comprehension is correlated with temporal response patterns recorded from auditory cortex , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[48]  Steven A. Hillyard,et al.  Identification of the neural sources of the pattern-reversal VEP , 2005, NeuroImage.

[49]  H B Barlow,et al.  Single units and sensation: a neuron doctrine for perceptual psychology? , 1972, Perception.

[50]  Denis Brunet,et al.  Topographic ERP Analyses: A Step-by-Step Tutorial Review , 2008, Brain Topography.

[51]  Barak A. Pearlmutter,et al.  The VESPA: A method for the rapid estimation of a visual evoked potential , 2006, NeuroImage.

[52]  Brian N. Pasley,et al.  Reconstructing Speech from Human Auditory Cortex , 2012, PLoS biology.

[53]  John J. Foxe,et al.  Isolating early cortical generators of visual-evoked activity: a systems identification approach , 2012, Experimental Brain Research.

[54]  Edmund C. Lalor,et al.  Low-Frequency Cortical Entrainment to Speech Reflects Phoneme-Level Processing , 2015, Current Biology.

[55]  Barak A. Pearlmutter,et al.  Dissecting the cellular contributions to early visual sensory processing deficits in schizophrenia using the VESPA evoked response , 2008, Schizophrenia Research.

[56]  D. D. Greenwood A cochlear frequency-position function for several species--29 years later. , 1990, The Journal of the Acoustical Society of America.

[57]  Barak A. Pearlmutter,et al.  Isolating endogenous visuo-spatial attentional effects using the novel visual-evoked spread spectrum analysis (VESPA) technique , 2007, The European journal of neuroscience.

[58]  K. Sen,et al.  Spectral-temporal Receptive Fields of Nonlinear Auditory Neurons Obtained Using Natural Sounds , 2022 .

[59]  D. Abrams,et al.  Right-Hemisphere Auditory Cortex Is Dominant for Coding Syllable Patterns in Speech , 2008, The Journal of Neuroscience.

[60]  Nima Mesgarani,et al.  Phoneme representation and classification in primary auditory cortex. , 2008, The Journal of the Acoustical Society of America.

[61]  Vasilis Z. Marmarelis,et al.  Nonlinear Dynamic Modeling of Physiological Systems , 2004 .

[62]  P Kuyper,et al.  Triggered correlation. , 1968, IEEE transactions on bio-medical engineering.

[63]  T. Picton Hearing in Time: Evoked Potential Studies of Temporal Processing , 2013, Ear and hearing.

[64]  Todd C. Handy,et al.  Event-related potentials : a methods handbook , 2005 .

[65]  D. Heeger,et al.  Linear Systems Analysis of Functional Magnetic Resonance Imaging in Human V1 , 1996, The Journal of Neuroscience.

[66]  John J. Foxe,et al.  Investigating the temporal dynamics of auditory cortical activation to silent lipreading , 2015, 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER).

[67]  Sethu Vijayakumar,et al.  Evaluation of regression methods for the continuous decoding of finger movement from surface EMG and accelerometry , 2015, 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER).

[68]  Michael J. Crosse,et al.  Eye Can Hear Clearly Now: Inverse Effectiveness in Natural Audiovisual Speech Processing Relies on Long-Term Crossmodal Temporal Integration , 2016, The Journal of Neuroscience.

[69]  T. Picton,et al.  Human Cortical Responses to the Speech Envelope , 2008, Ear and hearing.

[70]  Jonathan Z. Simon,et al.  Adaptive Temporal Encoding Leads to a Background-Insensitive Cortical Representation of Speech , 2013, The Journal of Neuroscience.

[71]  John J. Foxe,et al.  Neural responses to uninterrupted natural speech can be extracted with precise temporal resolution , 2010, The European journal of neuroscience.

[72]  John J. Foxe,et al.  Early Spatial Attentional Modulation of Inputs to the Fovea , 2010, The Journal of Neuroscience.

[73]  Richard Coppola,et al.  A SYSTEM TRANSFER FUNCTION FOR VISUAL EVOKED POTENTIALS , 1979 .

[74]  Sidney S. Simon,et al.  Merging of the Senses , 2008, Front. Neurosci..