A neurocomputational model of the N 400 and the P 600 in semantic processing

Ten years ago, researchers using event-related brain potentials (ERPs) to study language comprehension were puzzled by what looked like a Semantic Illusion: Semantically anomalous, but structurally well-formed sentences did not affect the N400 component—traditionally taken to reflect semantic integration—but instead produced a P600 effect, which is generally linked to syntactic processing. This finding led to a considerable amount of debate, and a number of complex processing models have been proposed as an explanation. What these models have in common is that they postulate two or more separate processing streams, in order to reconcile the Semantic Illusion and other semantically induced P600 effects with the traditional interpretations of the N400 and the P600. Recently, however, these multi-stream models have been called into question, and a simpler single-stream model has been proposed. According to this alternative model, the N400 component reflects the retrieval of word meaning from semantic memory, and the P600 component indexes the integration of this meaning into the unfolding utterance interpretation. In the present paper, we provide support for this “Retrieval–Integration (RI)” account by instantiating it as a neurocomputational model. This neurocomputational model is the first to successfully simulate the N400 and P600 amplitude in language comprehension, and simulations with this model provide a proof of concept of the single-stream RI account of semantically induced patterns of N400 and P600 modulations. Correspondence should be send to Harm Brouwer, Department of Language Science and Technology, Saarland University, Building C7.1, 66123 Saarbr€ucken, Germany. E-mail: brouwer@coli.uni-saarland.de This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.

[1]  David Badre,et al.  Left ventrolateral prefrontal cortex and the cognitive control of memory , 2007, Neuropsychologia.

[2]  Mathieu Koppen,et al.  Modeling knowledge-based inferences in story comprehension , 2003, Cogn. Sci..

[3]  M. Kutas,et al.  Psycholinguistics Electrified II (1994–2005) , 2006 .

[4]  M. Mesulam,et al.  From sensation to cognition. , 1998, Brain : a journal of neurology.

[5]  Harm Brouwer,et al.  The Electrophysiology of Language Comprehension , 2014 .

[6]  D. Poeppel,et al.  The cortical organization of speech processing , 2007, Nature Reviews Neuroscience.

[7]  David Caplan,et al.  Task-dependent and task-independent neurovascular responses to syntactic processing , 2008, Cortex.

[8]  D. Caplan,et al.  Electrophysiological distinctions in processing conceptual relationships within simple sentences. , 2003, Brain research. Cognitive brain research.

[9]  Mante S. Nieuwland,et al.  Testing the limits of the semantic illusion phenomenon: ERPs reveal temporary semantic change deafness in discourse comprehension. , 2005, Brain research. Cognitive brain research.

[10]  P. Morosan,et al.  Broca's Region: Novel Organizational Principles and Multiple Receptor Mapping , 2010, PLoS biology.

[11]  Angela D. Friederici,et al.  The language skeleton after dissecting meaning: A functional segregation within Broca's Area , 2015, NeuroImage.

[12]  Harm Brouwer,et al.  The electrophysiology of language comprehension A neurocomputational model , 2010 .

[13]  Marshall R. Mayberry,et al.  Situated sentence processing: The coordinated interplay account and a neurobehavioral model , 2010, Brain and Language.

[14]  Michael P Milham,et al.  Reading networks at rest. , 2010, Cerebral cortex.

[15]  Mante S. Nieuwland,et al.  Testing the limits of the semantic illusion phenomenon: ERPs reveal temporary change deafness in discourse comprehension , 2004 .

[16]  John C. J. Hoeks,et al.  A time and place for language comprehension: mapping the N400 and the P600 to a minimal cortical network , 2013, Front. Hum. Neurosci..

[17]  Kara D. Federmeier,et al.  Electrophysiology reveals semantic memory use in language comprehension , 2000, Trends in Cognitive Sciences.

[18]  John Boddy,et al.  Evoked potentials and the dynamics of language processing , 1981, Biological Psychology.

[19]  M. Schlesewsky,et al.  Reconciling time, space and function: A new dorsal–ventral stream model of sentence comprehension , 2013, Brain and Language.

[20]  Petra Hendriks,et al.  Questions Left Unanswered: How the Brain Responds to Missing Information , 2013, PloS one.

[21]  K. Berman,et al.  Human Dorsal and Ventral Auditory Streams Subserve Rehearsal-Based and Echoic Processes during Verbal Working Memory , 2005, Neuron.

[22]  P. Hagoort On Broca, brain, and binding: a new framework , 2005, Trends in Cognitive Sciences.

[23]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[24]  D. Pandya,et al.  The extreme capsule in humans and rethinking of the language circuitry , 2009, Brain Structure and Function.

[25]  Anders M. Dale,et al.  N400-like Magnetoencephalography Responses Modulated by Semantic Context, Word Frequency, and Lexical Class in Sentences , 2002, NeuroImage.

[26]  Milena Rabovsky,et al.  Simulating the N400 ERP component as semantic network error: Insights from a feature-based connectionist attractor model of word meaning , 2014, Cognition.

[27]  A. Friederici The cortical language circuit: from auditory perception to sentence comprehension , 2012, Trends in Cognitive Sciences.

[28]  William W. Graves,et al.  Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. , 2009, Cerebral cortex.

[29]  Ellen F. Lau,et al.  A cortical network for semantics: (de)constructing the N400 , 2008, Nature Reviews Neuroscience.

[30]  J. Trueswell,et al.  Cognitive control and parsing: Reexamining the role of Broca’s area in sentence comprehension , 2005, Cognitive, affective & behavioral neuroscience.

[31]  D. Plaut,et al.  A neurally plausible Parallel Distributed Processing model of Event-Related Potential word reading data , 2012, Brain and Language.

[32]  James L. McClelland,et al.  Sentence comprehension: A parallel distributed processing approach , 1989, Language and Cognitive Processes.

[33]  David Caplan,et al.  Syntactic and Thematic Constraint Effects on Blood Oxygenation Level Dependent Signal Correlates of Comprehension of Relative Clauses , 2008, Journal of Cognitive Neuroscience.

[34]  D. Poeppel,et al.  Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language , 2004, Cognition.

[35]  Angela D. Friederici,et al.  Pathways to language: fiber tracts in the human brain , 2009, Trends in Cognitive Sciences.

[36]  Mark S. Seidenberg,et al.  Semantic feature production norms for a large set of living and nonliving things , 2005, Behavior research methods.

[37]  John Hoeks,et al.  Electrophysiological Research on Conversation and Discourse Processing , 2014 .

[38]  Scott E. Fahlman,et al.  An empirical study of learning speed in back-propagation networks , 1988 .

[39]  Gina R. Kuperberg,et al.  Neural mechanisms of language comprehension: Challenges to syntax , 2007, Brain Research.

[40]  I. Rooij,et al.  Connectionist semantic systematicity , 2009, Cognition.

[41]  N. Dronkers,et al.  Lesion analysis of the brain areas involved in language comprehension , 2004, Cognition.

[42]  Franciska de Jong,et al.  TwNC: a Multifaceted Dutch News Corpus , 2007 .

[43]  Ina Bornkessel-Schlesewsky,et al.  Word order and Broca’s region: Evidence for a supra-syntactic perspective , 2009, Brain and Language.

[44]  Harm Brouwer,et al.  On the organization of the perisylvian cortex: Insights from the electrophysiology of language: Comment on "Towards a Computational Comparative Neuroprimatology: Framing the language-ready brain" by M.A. Arbib. , 2016, Physics of life reviews.

[45]  R. Cabeza,et al.  Imaging Cognition II: An Empirical Review of 275 PET and fMRI Studies , 2000, Journal of Cognitive Neuroscience.

[46]  C. C. Wood,et al.  Event-related potentials, lexical decision and semantic priming. , 1985, Electroencephalography and clinical neurophysiology.

[47]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[48]  S. Bookheimer Functional MRI of language: new approaches to understanding the cortical organization of semantic processing. , 2002, Annual review of neuroscience.

[49]  M M Mesulam,et al.  Large‐scale neurocognitive networks and distributed processing for attention, language, and memory , 1990, Annals of neurology.

[50]  H. Kolk,et al.  Structure and limited capacity in verbal working memory: A study with event-related potentials , 2003, Brain and Language.

[51]  C. Van Petten,et al.  Prediction during language comprehension: benefits, costs, and ERP components. , 2012, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[52]  A. Turken,et al.  The Neural Architecture of the Language Comprehension Network: Converging Evidence from Lesion and Connectivity Analyses , 2011, Front. Syst. Neurosci..

[53]  Bradley R. Buchsbaum,et al.  The Search for the Phonological Store: From Loop to Convolution , 2008, Journal of Cognitive Neuroscience.

[54]  Giosuè Baggio,et al.  The balance between memory and unification in semantics: A dynamic account of the N400 , 2011 .

[55]  Marshall R. Mayberry,et al.  Learning to Attend: A Connectionist Model of Situated Language Comprehension , 2009, Cogn. Sci..

[56]  Hartmut Fitz,et al.  Getting real about Semantic Illusions: Rethinking the functional role of the P600 in language comprehension , 2012, Brain Research.

[57]  Angela D. Friederici,et al.  Grounding language processing on basic neurophysiological principles , 2015, Trends in Cognitive Sciences.

[58]  Gregory Hickok,et al.  The Role of Broca's Area in Sentence Comprehension , 2011, Journal of Cognitive Neuroscience.

[59]  Peter Indefrey,et al.  Monitoring in language perception: Electrophysiological and hemodynamic responses to spelling violations , 2011, NeuroImage.

[60]  David C. Plaut,et al.  A connectionist model of sentence comprehension and production , 2002 .

[61]  Roger Levy,et al.  A noisy-channel model of rational human sentence comprehension under uncertain input , 2008, EMNLP 2008.

[62]  Douglas L. T. Rohde,et al.  An Improved Model of Semantic Similarity Based on Lexical Co-Occurrence , 2005 .

[63]  Keith A. Johnson,et al.  Cortical Hubs Revealed by Intrinsic Functional Connectivity: Mapping, Assessment of Stability, and Relation to Alzheimer's Disease , 2009, The Journal of Neuroscience.

[64]  Derek K. Jones,et al.  Perisylvian language networks of the human brain , 2005, Annals of neurology.

[65]  P. Hagoort Interplay between Syntax and Semantics during Sentence Comprehension: ERP Effects of Combining Syntactic and Semantic Violations , 2003, Journal of Cognitive Neuroscience.

[66]  Riitta Salmelin,et al.  Localization of Syntactic and Semantic Brain Responses using Magnetoencephalography , 2007, Journal of Cognitive Neuroscience.

[67]  Van Berkum,et al.  The neuropragmatics of 'simple' utterance comprehension: An ERP review , 2009 .

[68]  William D. Marslen-Wilson,et al.  Left inferior frontal cortex and syntax: function, structure and behaviour in patients with left hemisphere damage , 2011, Brain : a journal of neurology.

[69]  John C J Hoeks,et al.  Seeing words in context: the interaction of lexical and sentence level information during reading. , 2004, Brain research. Cognitive brain research.

[70]  Katrin Amunts,et al.  Architecture and organizational principles of Broca's region , 2012, Trends in Cognitive Sciences.

[71]  Cornelius Weiller,et al.  Please don’t underestimate the ventral pathway in language , 2009, Trends in Cognitive Sciences.

[72]  S. Kuriki,et al.  MEG study on neural activities associated with syntactic and semantic violations in spoken Korean sentences , 2005, Neuroscience Research.

[73]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[74]  C. Price The anatomy of language: a review of 100 fMRI studies published in 2009 , 2010, Annals of the New York Academy of Sciences.

[75]  Matthias Schlesewsky,et al.  An alternative perspective on “semantic P600” effects in language comprehension , 2008, Brain Research Reviews.

[76]  Colin M. Brown,et al.  The syntactic positive shift (sps) as an erp measure of syntactic processing , 1993 .

[77]  A. Friederici The brain basis of language processing: from structure to function. , 2011, Physiological reviews.

[78]  L. Osterhout,et al.  The independence of combinatory semantic processing: Evidence from event-related potentials , 2005 .

[79]  Volkmar Glauche,et al.  Ventral and dorsal pathways for language , 2008, Proceedings of the National Academy of Sciences.

[80]  Y. Grodzinsky,et al.  The battle for Broca’s region , 2008, Trends in Cognitive Sciences.