EEG decoding of spoken words in bilingual listeners: from words to language invariant semantic-conceptual representations

Spoken word recognition and production require fast transformations between acoustic, phonological, and conceptual neural representations. Bilinguals perform these transformations in native and non-native languages, deriving unified semantic concepts from equivalent, but acoustically different words. Here we exploit this capacity of bilinguals to investigate input invariant semantic representations in the brain. We acquired EEG data while Dutch subjects, highly proficient in English listened to four monosyllabic and acoustically distinct animal words in both languages (e.g., “paard”–“horse”). Multivariate pattern analysis (MVPA) was applied to identify EEG response patterns that discriminate between individual words within one language (within-language discrimination) and generalize meaning across two languages (across-language generalization). Furthermore, employing two EEG feature selection approaches, we assessed the contribution of temporal and oscillatory EEG features to our classification results. MVPA revealed that within-language discrimination was possible in a broad time-window (~50–620 ms) after word onset probably reflecting acoustic-phonetic and semantic-conceptual differences between the words. Most interestingly, significant across-language generalization was possible around 550–600 ms, suggesting the activation of common semantic-conceptual representations from the Dutch and English nouns. Both types of classification, showed a strong contribution of oscillations below 12 Hz, indicating the importance of low frequency oscillations in the neural representation of individual words and concepts. This study demonstrates the feasibility of MVPA to decode individual spoken words from EEG responses and to assess the spectro-temporal dynamics of their language invariant semantic-conceptual representations. We discuss how this method and results could be relevant to track the neural mechanisms underlying conceptual encoding in comprehension and production.

[1]  David Poeppel,et al.  Cortical oscillations and speech processing: emerging computational principles and operations , 2012, Nature Neuroscience.

[2]  Colin M. Brown,et al.  Electrophysiological Evidence for Early Contextual Influences during Spoken-Word Recognition: N200 Versus N400 Effects , 2001, Journal of Cognitive Neuroscience.

[3]  Paul Boersma,et al.  Praat: doing phonetics by computer , 2003 .

[4]  E. T. Possing,et al.  Human temporal lobe activation by speech and nonspeech sounds. , 2000, Cerebral cortex.

[5]  M. Pickering,et al.  An integrated theory of language production and comprehension. , 2013, The Behavioral and brain sciences.

[6]  Kayoko Okada,et al.  Conduction aphasia, sensory-motor integration, and phonological short-term memory – An aggregate analysis of lesion and fMRI data , 2011, Brain and Language.

[7]  Matthew A. Lambon Ralph,et al.  Differential Contributions of Bilateral Ventral Anterior Temporal Lobe and Left Anterior Superior Temporal Gyrus to Semantic Processes , 2011, Journal of Cognitive Neuroscience.

[8]  P. Hagoort.  The fractionation of spoken language understanding by measuring electrical and magnetic brain signals , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[9]  D. Poeppel,et al.  Phase Patterns of Neuronal Responses Reliably Discriminate Speech in Human Auditory Cortex , 2007, Neuron.

[10]  Terrence J. Sejnowski,et al.  Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources , 1999, Neural Computation.

[11]  Milene Bonte,et al.  Developmental changes in ERP correlates of spoken word recognition during early school years: a phonological priming study , 2004, Clinical Neurophysiology.

[12]  Rasha Abdel Rahman,et al.  Electrophysiological Chronometry of Semantic Context Effects in Language Production , 2011, Journal of Cognitive Neuroscience.

[13]  Kristin Lemhöfer,et al.  Introducing LexTALE: A quick and valid Lexical Test for Advanced Learners of English , 2011, Behavior research methods.

[14]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[15]  E. Plante,et al.  Time course of word identification and semantic integration in spoken language. , 1999, Journal of experimental psychology. Learning, memory, and cognition.

[16]  James M. McQueen,et al.  Decoding Speech Perception by Native and Non-Native Speakers Using Single-Trial Electrophysiological Data , 2013, PloS one.

[17]  Alexander G. Huth,et al.  Attention During Natural Vision Warps Semantic Representation Across the Human Brain , 2013, Nature Neuroscience.

[18]  A. Damasio,et al.  A neural basis for lexical retrieval , 1996, Nature.

[19]  A. Friederici,et al.  Auditory perception and syntactic cognition: brain activity‐based decoding within and across subjects , 2012, The European journal of neuroscience.

[20]  W. Marslen-Wilson.  Functional parallelism in spoken word-recognition , 1987, Cognition.

[21]  Björn Herrmann,et al.  Neural Oscillations in Speech: Don't be Enslaved by the Envelope , 2012, Front. Hum. Neurosci..

[22]  James L. McClelland,et al.  The TRACE model of speech perception , 1986, Cognitive Psychology.

[23]  Dirk Koester,et al.  Morphological priming in overt language production: Electrophysiological evidence from Dutch , 2008, NeuroImage.

[24]  Rainer Goebel,et al.  Predicting EEG single trial responses with simultaneous fMRI and Relevance Vector Machine regression , 2011, NeuroImage.

[25]  A. Damasio,et al.  Convergence and divergence in a neural architecture for recognition and memory , 2009, Trends in Neurosciences.

[26]  Eric Halgren,et al.  First-Pass Selectivity for Semantic Categories in Human Anteroventral Temporal Lobe , 2011, The Journal of Neuroscience.

[27]  Paula Breen,et al.  WHO IS TO SAY , 1967 .

[28]  Peter Indefrey,et al.  The Spatial and Temporal Signatures of Word Production Components: A Critical Update , 2011, Front. Psychology.

[29]  Giancarlo Valente,et al.  Multivariate analysis of fMRI time series: classification and regression of brain responses using machine learning. , 2008, Magnetic resonance imaging.

[30]  David Poeppel,et al.  Acoustic landmarks drive delta–theta oscillations to enable speech comprehension by facilitating perceptual parsing , 2014, NeuroImage.

[31]  Kara D. Federmeier,et al.  Electrophysiology reveals semantic memory use in language comprehension , 2000, Trends in Cognitive Sciences.

[32]  W. Levelt,et al.  Speaking: From Intention to Articulation , 1990 .

[33]  Riitta Salmelin,et al.  Tracking neural coding of perceptual and semantic features of concrete nouns , 2012, NeuroImage.

[34]  Matthew K. Leonard,et al.  Independence of early speech processing from word meaning. , 2013, Cerebral cortex.

[35]  Feng Rong,et al.  Sensorimotor Integration in Speech Processing: Computational Basis and Neural Organization , 2011, Neuron.

[36]  J. Obleser,et al.  Frequency modulation entrains slow neural oscillations and optimizes human listening behavior , 2012, Proceedings of the National Academy of Sciences.

[37]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[38]  R. Salmelin,et al.  Time course of top-down and bottom-up influences on syllable processing in the auditory cortex. , 2006, Cerebral cortex.

[39]  M. Besson,et al.  Behavioral and Electrophysiological Study of Phonological Priming between Bisyllabic Spoken Words , 2001, Journal of Cognitive Neuroscience.

[40]  R. Oostenveld,et al.  I see what you mean: Theta power increases are involved in the retrieval of lexical semantic information , 2008, Brain and Language.

[41]  A. Baddeley.  Working memory and language: an overview. , 2003, Journal of communication disorders.

[42]  Tom M. Mitchell,et al.  Commonality of neural representations of words and pictures , 2011, NeuroImage.

[43]  Matthew H. Davis,et al.  Neural Oscillations Carry Speech Rhythm through to Comprehension , 2012, Front. Psychology.

[44]  Peter Indefrey,et al.  The use of conceptual components in language production: an ERP study , 2014, Front. Psychol..

[45]  Elizabeth Jefferies,et al.  Both the Middle Temporal Gyrus and the Ventral Anterior Temporal Area Are Crucial for Multimodal Semantic Processing: Distortion-corrected fMRI Evidence for a Double Gradient of Information Convergence in the Temporal Lobes , 2012, Journal of Cognitive Neuroscience.

[46]  Giancarlo Valente,et al.  Dynamic and Task-Dependent Encoding of Speech and Voice by Phase Reorganization of Cortical Oscillations , 2009, The Journal of Neuroscience.

[47]  W. Levelt,et al.  The spatial and temporal signatures of word production components , 2004, Cognition.

[48]  Antoine J. Shahin,et al.  Brain oscillations during semantic evaluation of speech , 2009, Brain and Cognition.

[49]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[50]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[51]  Werner Sommer,et al.  Does phonological encoding in speech production always follow the retrieval of semantic knowledge? Electrophysiological evidence for parallel processing. , 2003, Brain research. Cognitive brain research.

[52]  M. Kutas,et al.  Electrophysiological estimates of the time course of semantic and phonological encoding during implicit picture naming. , 2000, Psychophysiology.

[53]  Ankoor S. Shah,et al.  An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex. , 2005, Journal of neurophysiology.

[54]  T. Sejnowski,et al.  Dynamic Brain Sources of Visual Evoked Responses , 2002, Science.

[55]  J. Schoffelen,et al.  Beta oscillations relate to the N400m during language comprehension , 2012, Human brain mapping.

[56]  Sean M. Polyn,et al.  Beyond mind-reading: multi-voxel pattern analysis of fMRI data , 2006, Trends in Cognitive Sciences.

[57]  Giancarlo Valente,et al.  Brain-Based Translation: fMRI Decoding of Spoken Words in Bilinguals Reveals Language-Independent Semantic Representations in Anterior Temporal Lobe , 2014, The Journal of Neuroscience.

[58]  B. Mesquita,et al.  Adjustment to Chronic Diseases and Terminal Illness Health Psychology : Psychological Adjustment to Chronic Disease , 2006 .

[59]  Valerie A. Carr,et al.  Spatiotemporal Dynamics of Modality-Specific and Supramodal Word Processing , 2003, Neuron.

[60]  R. H. Baayen,et al.  The CELEX Lexical Database (CD-ROM) , 1996 .

[61]  D. Poeppel,et al.  The cortical organization of speech processing , 2007, Nature Reviews Neuroscience.

[62]  Jonas Obleser,et al.  Attentional influences on functional mapping of speech sounds in human auditory cortex , 2004, BMC Neuroscience.

[63]  C. Waltz.  Validation study. , 1988, NLN publications.

[64]  C Van Petten,et al.  Time course of word identification and semantic integration in spoken language. , 1999, Journal of experimental psychology. Learning, memory, and cognition.

[65]  I. Christoffels,et al.  Language conflict in translation: An ERP study of translation production , 2013 .

[66]  Robert Oostenveld,et al.  Identifying Object Categories from Event-Related EEG: Toward Decoding of Conceptual Representations , 2010, PloS one.

[67]  Lisa D. Sanders,et al.  An ERP study of continuous speech processing. I. Segmentation, semantics, and syntax in native speakers. , 2003, Brain research. Cognitive brain research.

[68]  Lars Hausfeld,et al.  Pattern analysis of EEG responses to speech and voice: Influence of feature grouping , 2012, NeuroImage.

[69]  Rainer Goebel,et al.  "Who" Is Saying "What"? Brain-Based Decoding of Human Voice and Speech , 2008, Science.

[70]  T. Rogers,et al.  Where do you know what you know? The representation of semantic knowledge in the human brain , 2007, Nature Reviews Neuroscience.

[71]  Sydney S. Cash,et al.  Decoding word and category-specific spatiotemporal representations from MEG and EEG , 2011, NeuroImage.

[72]  E. Formisano,et al.  Auditory Cortex Encodes the Perceptual Interpretation of Ambiguous Sound , 2011, The Journal of Neuroscience.

[73]  R. Salmelin.  Clinical neurophysiology of language: The MEG approach , 2007, Clinical Neurophysiology.

[74]  Giancarlo Valente,et al.  Task-Dependent Decoding of Speaker and Vowel Identity from Auditory Cortical Response Patterns , 2014, The Journal of Neuroscience.

[75]  Marc Brysbaert,et al.  A validation study of the age-of-acquisition norms collected by Ghyselinck, De Moor, & Brysbaert , 2000 .

[76]  J. Rauschecker,et al.  Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing , 2009, Nature Neuroscience.

[77]  Bryan R. Conroy,et al.  A Common, High-Dimensional Model of the Representational Space in Human Ventral Temporal Cortex , 2011, Neuron.

[78]  Lucia Melloni,et al.  Brain Oscillations during Spoken Sentence Processing , 2012, Journal of Cognitive Neuroscience.

[79]  A. Hayes,et al.  Combining independent p values: extensions of the Stouffer and binomial methods. , 2000, Psychological methods.

[80]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[81]  P. Hagoort,et al.  Integration of Word Meaning and World Knowledge in Language Comprehension , 2004, Science.

[82]  R. Salmelin,et al.  Spatiotemporal interaction between sound form and meaning during spoken word perception. , 2008, Cerebral cortex.

[83]  Sonja A. Kotz,et al.  Alpha and theta brain oscillations index dissociable processes in spoken word recognition , 2014, NeuroImage.

[84]  Angela D. Friederici,et al.  Exploring the Activation of Semantic and Phonological Codes during Speech Planning with Event-Related Brain Potentials , 2002, Journal of Cognitive Neuroscience.

[85]  S. Scott,et al.  Identification of a pathway for intelligible speech in the left temporal lobe. , 2000, Brain : a journal of neurology.

[86]  Magdalena Szumilas.  Explaining odds ratios. , 2010, Journal of the Canadian Academy of Child and Adolescent Psychiatry = Journal de l'Academie canadienne de psychiatrie de l'enfant et de l'adolescent.