Separate streams or probabilistic inference? What the N400 can tell us about the comprehension of events

ABSTRACT Since the early 2000s, several event-related potential studies have challenged the assumption that we always use syntactic contextual information to influence semantic processing of incoming words, as reflected by the N400 component. One approach for explaining these findings is to posit distinct semantic and syntactic processing mechanisms, each with distinct time courses. While this approach can explain specific datasets, it cannot account for the wider body of findings. I propose an alternative explanation: a dynamic generative framework in which our goal is to infer the underlying event that best explains the set of inputs encountered at any given time. Within this framework, combinations of semantic and syntactic cues with varying reliabilities are used as evidence to weight probabilistic hypotheses about this event. I further argue that the computational principles of this framework can be extended to understand how we infer situation models during discourse comprehension, and intended messages during spoken communication.

[1]  James L. McClelland,et al.  An interactive activation model of context effects in letter perception: Part 2. The contextual enhancement effect and some tests and extensions of the model. , 1982, Psychological review.

[2]  Geoffrey E. Hinton Learning multiple layers of representation , 2007, Trends in Cognitive Sciences.

[3]  W. Kintsch The role of knowledge in discourse comprehension: a construction-integration model. , 1988, Psychological review.

[4]  J. Saunders,et al.  Do humans optimally integrate stereo and texture information for judgments of surface slant? , 2003, Vision Research.

[5]  S. Kakade,et al.  Learning and selective attention , 2000, Nature Neuroscience.

[6]  Karl J. Friston,et al.  Attention, Uncertainty, and Free-Energy , 2010, Front. Hum. Neurosci..

[7]  Michael P. Wilson,et al.  Making simple sentences hard: Verb bias effects in simple direct object sentences. , 2009, Journal of memory and language.

[8]  Kara D. Federmeier,et al.  Right hemisphere sensitivity to word- and sentence-level context: evidence from event-related brain potentials. , 2005, Journal of Experimental Psychology. Learning, Memory and Cognition.

[9]  Kathryn Bock,et al.  Language production : Grammatical encoding , 1994 .

[10]  Marta Kutas,et al.  Comprehending how visual context influences incremental sentence processing: Insights from ERPs and picture-sentence verification. , 2011, Psychophysiology.

[11]  Gary S. Dell,et al.  Mechanisms for Listener-Adaptation in Language Production: Limiting the Role of the "Model of the Listener" , 2013 .

[12]  J. Tenenbaum,et al.  A tutorial introduction to Bayesian models of cognitive development , 2011, Cognition.

[13]  Gina R. Kuperberg,et al.  Neural mechanisms of language comprehension: Challenges to syntax , 2007, Brain Research.

[14]  Ellen F. Lau,et al.  A cortical network for semantics: (de)constructing the N400 , 2008, Nature Reviews Neuroscience.

[15]  Martin Paczynski,et al.  Establishing Causal Coherence across Sentences: An ERP Study , 2011, Journal of Cognitive Neuroscience.

[16]  L. Stowe Thematic Structures and Sentence Comprehension , 1989 .

[17]  W. Marslen-Wilson Functional parallelism in spoken word-recognition , 1987, Cognition.

[18]  S. Levinson Action formation and ascription , 2013 .

[19]  D. Caplan,et al.  The role of animacy and thematic relationships in processing active English sentences: Evidence from event-related potentials , 2007, Brain and Language.

[20]  L. Osterhout,et al.  The independence of combinatory semantic processing: Evidence from event-related potentials , 2005 .

[21]  John K. Krusch Bayesian approaches to associative learning: From passive to active learning , 2008 .

[22]  J. Fodor The Modularity of mind. An essay on faculty psychology , 1986 .

[23]  Richard N. Aslin,et al.  Learning to Represent a Multi-Context Environment: More than Detecting Changes , 2012, Front. Psychology.

[24]  John C J Hoeks,et al.  Seeing words in context: the interaction of lexical and sentence level information during reading. , 2004, Brain research. Cognitive brain research.

[25]  J. Fodor,et al.  The structure of a semantic theory , 1963 .

[26]  J. Grafman,et al.  Human prefrontal cortex: processing and representational perspectives , 2003, Nature Reviews Neuroscience.

[27]  Susan M. Garnsey Event-related brain potentials in the study of language , 1993 .

[28]  A. Staub,et al.  The influence of cloze probability and item constraint on cloze task response time , 2015 .

[29]  P. Schwanenflugel,et al.  Semantic relatedness and the scope of facilitation for upcoming words in sentences. , 1988 .

[30]  C. Clifton,et al.  The independence of syntactic processing , 1986 .

[31]  Anthony J. Sanford On the nature of text-driven inference. , 1990 .

[32]  Colin Phillips,et al.  A “bag-of-arguments” mechanism for initial verb predictions , 2016 .

[33]  Kara D. Federmeier,et al.  Time for prediction? The effect of presentation rate on predictive sentence comprehension during word-by-word reading , 2015, Cortex.

[34]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[35]  Marc Brysbaert,et al.  Introducing a new entity into discourse: comprehension and production evidence for the status of Dutch er "there" as a higher-level expectancy monitor. , 2009, Acta psychologica.

[36]  A. Clark Whatever next? Predictive brains, situated agents, and the future of cognitive science. , 2013, The Behavioral and brain sciences.

[37]  Fernanda Ferreira,et al.  The misinterpretation of noncanonical sentences , 2003, Cognitive Psychology.

[38]  Megan A. Boudewyn,et al.  Processing new and repeated names: Effects of coreference on repetition priming with speech and fast RSVP , 2007, Brain Research.

[39]  Todd R. Ferretti,et al.  Thematic Roles as Verb-specific Concepts , 1997 .

[40]  C. Petten A comparison of lexical and sentence-level context effects in event-related potentials , 1993 .

[41]  Katherine A. DeLong,et al.  Probabilistic word pre-activation during language comprehension inferred from electrical brain activity , 2005, Nature Neuroscience.

[42]  G. Kuperberg,et al.  Multiple Influences of Semantic Memory on Sentence Processing: Distinct Effects of Semantic Relatedness on Violations of Real-World Event/State Knowledge and Animacy Selection Restrictions. , 2012, Journal of memory and language.

[43]  Mante S. Nieuwland,et al.  Great expectations: Specific lexical anticipation influences the processing of spoken language , 2007, BMC Neuroscience.

[44]  D. Caplan,et al.  Electrophysiological distinctions in processing conceptual relationships within simple sentences. , 2003, Brain research. Cognitive brain research.

[45]  Christoph Scheepers,et al.  Event-based plausibility immediately influences on-line language comprehension. , 2011, Journal of experimental psychology. Learning, memory, and cognition.

[46]  Angela D. Friederici,et al.  Mapping sentence form onto meaning: The syntax–semantic interface , 2007, Brain Research.

[47]  Kara D. Federmeier,et al.  Multiple effects of sentential constraint on word processing , 2007, Brain Research.

[48]  J. Kruschke Bayesian approaches to associative learning: From passive to active learning , 2008, Learning & behavior.

[49]  M. Kutas,et al.  Event-related brain potentials to semantically inappropriate and surprisingly large words , 1980, Biological Psychology.

[50]  Mante S. Nieuwland,et al.  When Peanuts Fall in Love: N400 Evidence for the Power of Discourse , 2005, Journal of Cognitive Neuroscience.

[51]  Jason Bohan,et al.  Anomalies at the Borderline of Awareness: An ERP Study , 2011, Journal of Cognitive Neuroscience.

[52]  Jerome L. Myers,et al.  Accessing the discourse representation during reading , 1998 .

[53]  C. Fillmore Chapter 10 Frame semantics , 2006 .

[54]  Michael J. Spivey,et al.  Syntactic ambiguity resolution in discourse: modeling the effects of referential context and lexical frequency. , 1998, Journal of experimental psychology. Learning, memory, and cognition.

[55]  Dave F. Kleinschmidt,et al.  Robust speech perception: recognize the familiar, generalize to the similar, and adapt to the novel. , 2015, Psychological review.

[56]  Jeffrey L. Elman,et al.  Cues, Constraints, and Competition in Sentence Processing , 2004 .

[57]  Susan M. Garnsey,et al.  Semantic Influences On Parsing: Use of Thematic Role Information in Syntactic Ambiguity Resolution , 1994 .

[58]  Ken McRae,et al.  People Use their Knowledge of Common Events to Understand Language, and Do So as Quickly as Possible , 2009, Lang. Linguistics Compass.

[59]  J. Elman,et al.  Generalized event knowledge activation during online sentence comprehension. , 2012, Journal of memory and language.

[60]  Christopher T. Kello,et al.  Verb-specific constraints in sentence processing: separating effects of lexical preference from garden-paths. , 1993, Journal of experimental psychology. Learning, memory, and cognition.

[61]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[62]  Kathryn Bock,et al.  Exploring Levels of Processing in Sentence Production , 1987 .

[63]  Karl J. Friston,et al.  Active inference and epistemic value , 2015, Cognitive neuroscience.

[64]  Matthias Schlesewsky,et al.  The extended argument dependency model: a neurocognitive approach to sentence comprehension across languages. , 2006, Psychological review.

[65]  Karl J. Friston,et al.  A theory of cortical responses , 2005, Philosophical Transactions of the Royal Society B: Biological Sciences.

[66]  B. MacWhinney,et al.  Cue validity and sentence interpretation in English, German, and Italian , 1984 .

[67]  Walter Kintsch,et al.  Predication , 2001, Cogn. Sci..

[68]  M. Kutas,et al.  An electrophysiological analysis of animacy effects in the processing of object relative sentences. , 1999, Psychophysiology.

[69]  M. Kutas,et al.  Brain potentials during reading reflect word expectancy and semantic association , 1984, Nature.

[70]  R. Jacobs What determines visual cue reliability? , 2002, Trends in Cognitive Sciences.

[71]  William D. Marslen-Wilson,et al.  Lexical Representations in Spoken Language Comprehension , 1988 .

[72]  Roger C. Schank,et al.  Scripts, plans, goals and understanding: an inquiry into human knowledge structures , 1978 .

[73]  Yijin Wu,et al.  The Handbook of Conversation Analysis , 2015 .

[74]  Miriam R. L. Petruck FRAME SEMANTICS , 1996 .

[75]  Janet D. Fodor,et al.  The sausage machine: A new two-stage parsing model , 1978, Cognition.

[76]  Y. Niv,et al.  Exploring a latent cause theory of classical conditioning , 2012, Learning & Behavior.

[77]  Rolf A. Zwaan,et al.  Situation models in language comprehension and memory. , 1998, Psychological bulletin.

[78]  Ina Bornkessel-Schlesewsky,et al.  Meaningful physical changes mediate lexical–semantic integration: Top-down and form-based bottom-up information sources interact in the N400 , 2011, Neuropsychologia.

[79]  Noam Chomsky,et al.  वाक्यविन्यास का सैद्धान्तिक पक्ष = Aspects of the theory of syntax , 1965 .

[80]  P. Gordon,et al.  The interplay of discourse congruence and lexical association during sentence processing: Evidence from ERPs and eye tracking. , 2007, Journal of memory and language.

[81]  R. Jackendoff The Status of Thematic Relations in Linguistic Theory , 1987 .

[82]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[83]  David R. Dowty On the Semantic Content of the Notion of ‘Thematic Role’ , 1989 .

[84]  R. Levy Expectation-based syntactic comprehension , 2008, Cognition.

[85]  Mary Hare,et al.  Understanding and producing the reduced relative construction: Evidence from ratings, editing and corpora. , 2007, Journal of memory and language.

[86]  G. Kuperberg,et al.  Reversing expectations during discourse comprehension , 2015, Language, cognition and neuroscience.

[87]  Kara D. Federmeier,et al.  So that's what you meant! Event-related potentials reveal multiple aspects of context use during construction of message-level meaning , 2012, NeuroImage.

[88]  Karl G. D. Bailey,et al.  Good-Enough Representations in Language Comprehension , 2002 .

[89]  J. Elman,et al.  Effects of event knowledge in processing verbal arguments. , 2010, Journal of memory and language.

[90]  Wendy Bergervoet,et al.  Monitoring in language perception , 2017 .

[91]  John B. Black,et al.  Scripts in memory for text , 1979, Cognitive Psychology.

[92]  A. Sanford,et al.  Depth of processing in language comprehension: not noticing the evidence , 2002, Trends in Cognitive Sciences.

[93]  J. Toomasian The Case for the Case , 2016, Perfusion.

[94]  Charles F. Hockett,et al.  A mathematical theory of communication , 1948, MOCO.

[95]  D. Donaldson,et al.  It’s the way that you, er, say it: Hesitations in speech affect language comprehension , 2007, Cognition.

[96]  Gina R Kuperberg,et al.  Electrophysiological evidence for use of the animacy hierarchy, but not thematic role assignment, during verb-argument processing , 2011, Language and cognitive processes.

[97]  Milena Rabovsky,et al.  Simulating the N400 ERP component as semantic network error: Insights from a feature-based connectionist attractor model of word meaning , 2014, Cognition.

[98]  Alex B. Fine,et al.  A belief-updating model of adaptation and cue combination in syntactic comprehension , 2012, CogSci.

[99]  Lyn Frazier,et al.  Sentence processing: A tutorial review. , 1987 .

[100]  Lyn Frazier,et al.  The interaction of syntax and semantics during sentence processing: eye movements in the analysis of semantically biased sentences , 1983 .

[101]  Gina R. Kuperberg,et al.  A Hierarchical Generative Framework of Language Processing: Linking Language Perception, Interpretation, and Production Abnormalities in Schizophrenia , 2015, Front. Hum. Neurosci..

[102]  Ellen F. Lau,et al.  Dissociating N400 Effects of Prediction from Association in Single-word Contexts , 2013, Journal of Cognitive Neuroscience.

[103]  Dennis Norris,et al.  Word recognition: Context effects without priming , 1986, Cognition.

[104]  R C Burgess,et al.  Event related potentials I , 1992, Clinical Neurophysiology.

[105]  P. Resnik Selection and information: a class-based approach to lexical relationships , 1993 .

[106]  Van Berkum,et al.  The neuropragmatics of 'simple' utterance comprehension: An ERP review , 2009 .

[107]  Rajesh P. N. Rao,et al.  Bayesian brain : probabilistic approaches to neural coding , 2006 .

[108]  K. Forster Priming and the Effects of Sentence and Lexical Contexts on Naming Time: Evidence for Autonomous Lexical Processing* , 1981 .

[109]  H. Kolk,et al.  Structure and limited capacity in verbal working memory: A study with event-related potentials , 2003, Brain and Language.

[110]  J. Fodor,et al.  The Modularity of Mind: An Essay on Faculty Psychology , 1984 .

[111]  Susan M. Garnsey,et al.  The Contributions of Verb Bias and Plausibility to the Comprehension of Temporarily Ambiguous Sentences , 1997 .

[112]  P. Hagoort,et al.  Integration of Word Meaning and World Knowledge in Language Comprehension , 2004, Science.

[113]  Peter,et al.  Semantic Unification , 2008 .

[114]  Simon Garrod,et al.  The role of scenario mapping in text comprehension. , 1998 .

[115]  Joshua K. Hartshorne,et al.  The causes and consequences explicit in verbs , 2015, Language, cognition and neuroscience.

[116]  Peter Hagoort,et al.  The Neural Integration of Speaker and Message , 2008, Journal of Cognitive Neuroscience.

[117]  Hartmut Fitz,et al.  Getting real about Semantic Illusions: Rethinking the functional role of the P600 in language comprehension , 2012, Brain Research.

[118]  Robert A Jacobs,et al.  Bayesian learning theory applied to human cognition. , 2011, Wiley interdisciplinary reviews. Cognitive science.

[119]  J. Fodor The Modularity of mind. An essay on faculty psychology , 1986 .

[120]  G. Kuperberg,et al.  Friendly drug-dealers and terrifying puppies: Affective primacy can attenuate the N400 effect in emotional discourse contexts , 2013, Cognitive, affective & behavioral neuroscience.

[121]  Phillip J. Holcomb,et al.  Neural correlates of processing syntactic, semantic, and thematic relationships in sentences , 2006 .

[122]  Mante S. Nieuwland,et al.  On the incrementality of pragmatic processing: An ERP investigation of informativeness and pragmatic abilities. , 2010, Journal of memory and language.

[123]  L. Parkinson Thirty years and counting , 2013, Australasian journal on ageing.

[124]  Katherine A. DeLong,et al.  Quantifiers are incrementally interpreted in context, more than less. , 2015, Journal of memory and language.

[125]  M. Tanenhaus Afterword The impact of “The cognitive basis for linguistic structures” , 2013 .

[126]  M. Tanenhaus,et al.  Modeling the Influence of Thematic Fit (and Other Constraints) in On-line Sentence Comprehension , 1998 .

[127]  A. Hollingworth,et al.  Thematic Roles Assigned along the Garden Path Linger , 2001, Cognitive Psychology.

[128]  M. Tomasello,et al.  Beyond nature-nurture: Essays in honor of Elizabeth Bates , 2005 .

[129]  James L. McClelland,et al.  An interactive activation model of context effects in letter perception: I. An account of basic findings. , 1981 .

[130]  Maryellen C. MacDonald,et al.  The lexical nature of syntactic ambiguity resolution , 1994 .

[131]  Jeffrey Gruber Studies in lexical relations , 1965 .

[132]  Jeffrey M. Zacks,et al.  Event perception: a mind-brain perspective. , 2007, Psychological bulletin.

[133]  Marta Kutas,et al.  Verb aspect and the activation of event knowledge. , 2007, Journal of experimental psychology. Learning, memory, and cognition.

[134]  M. Coltheart Attention and Performance XII: The Psychology of Reading , 1987 .

[135]  S. Frisch,et al.  The N400 reflects problems of thematic hierarchizing , 2001, Neuroreport.

[136]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.

[137]  Laurie E. Cutting,et al.  Unraveling reading comprehension : behavioral, neurobiological, and genetic components , 2013 .

[138]  David S. Touretzky,et al.  Similarity and Discrimination in Classical Conditioning: A Latent Variable Account , 2004, NIPS.

[139]  Mante S. Nieuwland,et al.  When the Truth Is Not Too Hard to Handle , 2008, Psychological science.

[140]  Colin M. Brown,et al.  Semantic Integration in Sentences and Discourse: Evidence from the N400 , 1999, Journal of Cognitive Neuroscience.

[141]  Gina R Kuperberg,et al.  What do we mean by prediction in language comprehension? , 2016, Language, cognition and neuroscience.

[142]  Ina Bornkessel-Schlesewsky,et al.  The Role of Prominence Information in the Real-Time Comprehension of Transitive Constructions: A Cross-Linguistic Approach , 2009, Lang. Linguistics Compass.

[143]  Kara D. Federmeier,et al.  A Rose by Any Other Name: Long-Term Memory Structure and Sentence Processing , 1999 .

[144]  M. Tanenhaus,et al.  Context effects in syntactic ambiguity resolution: discourse and semantic influences in parsing reduced relative clauses. , 1993, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[145]  R. Shillcock,et al.  Rethinking the Word Frequency Effect: The Neglected Role of Distributional Information in Lexical Processing , 2001, Language and speech.