Time-domain analysis of neural tracking of hierarchical linguistic structures

Abstract When listening to continuous speech, cortical activity measured by MEG concurrently follows the rhythms of multiple linguistic structures, e.g., syllables, phrases, and sentences. This phenomenon was previously characterized in the frequency domain. Here, we investigate the waveform of neural activity tracking linguistic structures in the time domain and quantify the coherence of neural response phases over subjects listening to the same stimulus. These analyses are achieved by decomposing the multi‐channel MEG recordings into components that maximize the correlation between neural response waveforms across listeners. Each MEG component can be viewed as the recording from a virtual sensor that is spatially tuned to a cortical network showing coherent neural activity over subjects. This analysis reveals information not available from previous frequency‐domain analysis of MEG global field power: First, concurrent neural tracking of hierarchical linguistic structures emerges at the beginning of the stimulus, rather than slowly building up after repetitions of the same sentential structure. Second, neural tracking of the sentential structure is reflected by slow neural fluctuations, rather than, e.g., a series of short‐lasting transient responses at sentential boundaries. Lastly and most importantly, it shows that the MEG responses tracking the syllabic rhythm are spatially separable from the MEG responses tracking the sentential and phrasal rhythms. HighlightsNeural tracking of different linguistic levels in cortical networks separable by MEG.Early emergence of neural tracking of phrasal/sentential structures.A novel method to align and visualize the waveform of entrained neural activity.

[1]  Nicholas I. Fisher,et al.  Statistical Analysis of Circular Data , 1993 .

[2]  Edmund C. Lalor,et al.  Low-Frequency Cortical Entrainment to Speech Reflects Phoneme-Level Processing , 2015, Current Biology.

[3]  S. Garrod,et al.  Brain-to-brain coupling: a mechanism for creating and sharing a social world , 2012, Trends in Cognitive Sciences.

[4]  S. Shamma,et al.  Interaction between Attention and Bottom-Up Saliency Mediates the Representation of Foreground and Background in an Auditory Scene , 2009, PLoS biology.

[5]  Riitta Hari,et al.  Intersubject consistency of cortical MEG signals during movie viewing , 2014, NeuroImage.

[6]  J. Obleser,et al.  Frequency modulation entrains slow neural oscillations and optimizes human listening behavior , 2012, Proceedings of the National Academy of Sciences.

[7]  S. Dehaene,et al.  Cortical representation of the constituent structure of sentences , 2011, Proceedings of the National Academy of Sciences.

[8]  Terence W. Picton,et al.  Temporal integration in the human auditory cortex as represented by the development of the steady-state magnetic field , 2002, Hearing Research.

[9]  Lucas C. Parra,et al.  Joint decorrelation, a versatile tool for multichannel data analysis , 2014, NeuroImage.

[10]  J. Fodor,et al.  The active use of grammar in speech perception , 1966 .

[11]  P. Schyns,et al.  Speech Rhythms and Multiplexed Oscillatory Sensory Coding in the Human Brain , 2013, PLoS biology.

[12]  P. Sprent,et al.  Statistical Analysis of Circular Data. , 1994 .

[13]  David Poeppel,et al.  Interpretations of Frequency Domain Analyses of Neural Entrainment: Periodicity, Fundamental Frequency, and Harmonics , 2016, Front. Hum. Neurosci..

[14]  Lucia Melloni,et al.  Brain Oscillations during Spoken Sentence Processing , 2012, Journal of Cognitive Neuroscience.

[15]  John S. Johnson,et al.  Audience preferences are predicted by temporal reliability of neural processing , 2014, Nature Communications.

[16]  Wei Wu,et al.  Frequency Recognition Based on Canonical Correlation Analysis for SSVEP-Based BCIs , 2006, IEEE Transactions on Biomedical Engineering.

[17]  Thomas G. Bever,et al.  Sentence Comprehension: The Integration of Habits and Rules , 2001 .

[18]  Jonathan Z. Simon,et al.  Denoising based on spatial filtering , 2008, Journal of Neuroscience Methods.

[19]  C. Honey,et al.  Hierarchical process memory: memory as an integral component of information processing , 2015, Trends in Cognitive Sciences.

[20]  Jonathan Z. Simon,et al.  Abstract Journal of Neuroscience Methods 165 (2007) 297–305 Denoising based on time-shift PCA , 2007 .

[21]  J. Simon,et al.  Emergence of neural encoding of auditory objects while listening to competing speakers , 2012, Proceedings of the National Academy of Sciences.

[22]  J. Simon,et al.  Neural coding of continuous speech in auditory cortex during monaural and dichotic listening. , 2012, Journal of neurophysiology.

[23]  R. Hari,et al.  Brain basis of human social interaction: from concepts to brain imaging. , 2009, Physiological reviews.

[24]  Chong-sun Kim Canonical Analysis of Several Sets of Variables , 1973 .

[25]  Samuel Kaski,et al.  Identifying fragments of natural speech from the listener's MEG signals , 2013, Human brain mapping.

[26]  Ignacio Santamaría,et al.  A learning algorithm for adaptive canonical correlation analysis of several data sets , 2007, Neural Networks.

[27]  Iiro P. Jääskeläinen,et al.  Combined MEG and EEG show reliable patterns of electromagnetic brain activity during natural viewing , 2015, NeuroImage.

[28]  Xiaorong Gao,et al.  An online multi-channel SSVEP-based brain–computer interface using a canonical correlation analysis method , 2009, Journal of neural engineering.

[29]  M. S. John,et al.  MASTER: a Windows program for recording multiple auditory steady-state responses , 2000, Comput. Methods Programs Biomed..

[30]  Line Garnero,et al.  Inter-Brain Synchronization during Social Interaction , 2010, PloS one.

[31]  D. Poeppel,et al.  Phase Patterns of Neuronal Responses Reliably Discriminate Speech in Human Auditory Cortex , 2007, Neuron.

[32]  Antoine J. Shahin,et al.  Attentional Gain Control of Ongoing Cortical Speech Representations in a “Cocktail Party” , 2010, The Journal of Neuroscience.

[33]  Noam Chomsky,et al.  Structures, Not Strings: Linguistics as Part of the Cognitive Sciences , 2015, Trends in Cognitive Sciences.

[34]  D. Poeppel,et al.  Cortical Tracking of Hierarchical Linguistic Structures in Connected Speech , 2015, Nature Neuroscience.

[35]  Marco Buiatti,et al.  Investigating the neural correlates of continuous speech computation with frequency-tagged neuroelectric responses , 2009, NeuroImage.

[36]  C. Honey,et al.  Topographic Mapping of a Hierarchy of Temporal Receptive Windows Using a Narrated Story , 2011, The Journal of Neuroscience.

[37]  C. Schroeder,et al.  Low-frequency neuronal oscillations as instruments of sensory selection , 2009, Trends in Neurosciences.

[38]  Xingyu Wang,et al.  Frequency Recognition in SSVEP-Based BCI using Multiset Canonical Correlation Analysis , 2013, Int. J. Neural Syst..

[39]  G. Karmos,et al.  Entrainment of Neuronal Oscillations as a Mechanism of Attentional Selection , 2008, Science.

[40]  Jonathan Z. Simon,et al.  Fully complex magnetoencephalography , 2005, Journal of Neuroscience Methods.

[41]  Jonathan Z. Simon,et al.  Adaptive Temporal Encoding Leads to a Background-Insensitive Cortical Representation of Speech , 2013, The Journal of Neuroscience.

[42]  D. Heeger,et al.  Slow Cortical Dynamics and the Accumulation of Information over Long Timescales , 2012, Neuron.

[43]  Robin A A Ince,et al.  Irregular Speech Rate Dissociates Auditory Cortical Entrainment, Evoked Responses, and Frontal Alpha , 2015, The Journal of Neuroscience.

[44]  R. Malach,et al.  Intersubject Synchronization of Cortical Activity During Natural Vision , 2004, Science.