Can Peanuts Fall in Love with Distributional Semantics?

Context changes expectations about upcoming words - following a story involving an anthropomorphic peanut, comprehenders expect the sentence the peanut was in love more than the peanut was salted, as indexed by N400 amplitude (Nieuwland&van Berkum, 2006). This updating of expectations has been explained using Situation Models - mental representations of a described event. However, recent work showing that N400 amplitude is predictable from distributional information alone raises the question whether situation models are necessary for these contextual effects. We model the results of Nieuwland and van Berkum (2006) using six computational language models and three sets of word vectors, none of which have explicit situation models or semantic grounding. We find that a subset of these can fully model the effect found by Nieuwland and van Berkum (2006). Thus, at least some processing effects normally explained through situation models may not in fact require explicit situation models.

[1]  Megan D. Bardolph,et al.  Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects , 2023, Neurobiology of language.

[2]  James A. Michaelov,et al.  Collateral facilitation in humans and language models , 2022, CONLL.

[3]  Kara D. Federmeier,et al.  Context-based facilitation of semantic access follows both logarithmic and linear functions of stimulus probability , 2021, Journal of memory and language.

[4]  James A. Michaelov,et al.  So Cloze Yet So Far: N400 Amplitude Is Better Predicted by Distributional Information Than Human Predictability Judgements , 2021, IEEE Transactions on Cognitive and Developmental Systems.

[5]  Benjamin K. Bergen,et al.  Different kinds of cognitive plausibility: why are transformers better than RNNs at predicting N400 amplitude? , 2021, CogSci.

[6]  Stella Biderman,et al.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow , 2021 .

[7]  S. Frank,et al.  Human Sentence Processing: Recurrence or Attention? , 2020, CMCL.

[8]  Peter Ford Dominey,et al.  A Model of Online Temporal-Spatial Integration for Immediacy and Overrule in Discourse Comprehension , 2020, Neurobiology of Language.

[9]  Benjamin K. Bergen,et al.  How well does surprisal explain N400 amplitude under different experimental conditions? , 2020, CONLL.

[10]  Julia Taylor Rayz,et al.  Exploring BERT’s sensitivity to lexical cues using tests from semantic priming , 2020, FINDINGS.

[11]  Bettina Berendt,et al.  RobBERT: a Dutch RoBERTa-based Language Model , 2020, FINDINGS.

[12]  Hinrich Schütze,et al.  Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly , 2019, ACL.

[13]  Joel Nothman,et al.  SciPy 1.0-Fundamental Algorithms for Scientific Computing in Python , 2019, ArXiv.

[14]  Yuji Matsumoto,et al.  Wikipedia2Vec: An Efficient Toolkit for Learning and Visualizing the Embeddings of Words and Entities from Wikipedia , 2018, EMNLP.

[15]  Tommaso Caselli,et al.  BERTje: A Dutch BERT Model , 2019, ArXiv.

[16]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[17]  Kohske Takahashi,et al.  Welcome to the Tidyverse , 2019, J. Open Source Softw..

[18]  K. Dijkstra,et al.  Situation model updating in young and older adults , 2019, International Journal of Behavioral Development.

[19]  Stefan L. Frank,et al.  Evaluating information-theoretic measures of word prediction in naturalistic sentence reading , 2019, Neuropsychologia.

[20]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[21]  Franklin Chang,et al.  Language ERPs reflect learning through prediction error propagation , 2019, Cognitive Psychology.

[22]  Matthew W. Crocker,et al.  Expectation-based Comprehension: Modeling the Interaction of World Knowledge and Linguistic Experience , 2019 .

[23]  Stefan Frank,et al.  Comparing Gated and Simple Recurrent Neural Network Architectures as Models of Human Sentence Processing , 2018, CogSci.

[24]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[25]  Rémi Louf,et al.  Transformers : State-ofthe-art Natural Language Processing , 2019 .

[26]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[27]  Walter Kintsch,et al.  Revisiting the Construction—Integration Model of Text Comprehension and its Implications for Instruction , 2018, Theoretical Models and Processes of Literacy.

[28]  Gina R. Kuperberg,et al.  A Tale of Two Positivities (and the N400): Distinct neural signatures are evoked by confirmed and violated predictions at different levels of representation , 2018, bioRxiv.

[29]  James L. McClelland,et al.  Modelling the N400 brain potential as change in a probabilistic representation of meaning , 2018, Nature Human Behaviour.

[30]  Prakhar Gupta,et al.  Learning Word Vectors for 157 Languages , 2018, LREC.

[31]  Tomas Mikolov,et al.  Advances in Pre-Training Distributed Word Representations , 2017, LREC.

[32]  Per B. Brockhoff,et al.  lmerTest Package: Tests in Linear Mixed Effects Models , 2017 .

[33]  Matthew W. Crocker,et al.  A Neurocomputational Model of the N400 and the P600 in Language Processing , 2016, Cognitive science.

[34]  Steven G. Luke,et al.  Limits on lexical prediction during reading , 2016, Cognitive Psychology.

[35]  Walter Daelemans,et al.  Evaluating Unsupervised Dutch Word Embeddings as a Linguistic Resource , 2016, LREC.

[36]  Rolf A. Zwaan Situation models, mental simulations, and abstract concepts in discourse comprehension , 2015, Psychonomic bulletin & review.

[37]  Allyson Ettinger,et al.  Modeling N400 amplitude using vector space models of word representation , 2016, CogSci.

[38]  S. Frank,et al.  The ERP response to the amount of information conveyed by words in sentences , 2015, Brain and Language.

[39]  G. Kuperberg,et al.  Reversing expectations during discourse comprehension , 2015, Language, cognition and neuroscience.

[40]  Marta Kutas,et al.  Pre-Processing in Sentence Comprehension: Sensitivity to Likely Upcoming Meaning and Structure , 2014, Lang. Linguistics Compass.

[41]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[42]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[43]  Rolf A. Zwaan Embodiment and language comprehension: reframing the discussion , 2014, Trends in Cognitive Sciences.

[44]  Viviane Deprez,et al.  Action relevance in linguistic context drives word-induced motor activity , 2014, Front. Hum. Neurosci..

[45]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[46]  C. Van Petten,et al.  Examining the N400 semantic context effect item-by-item: relationship to corpus-based measures of word co-occurrence. , 2014, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[47]  Kerstin Fischer,et al.  Beyond the sentence , 2013 .

[48]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[49]  Ellen F. Lau,et al.  Dissociating N400 Effects of Prediction from Association in Single-word Contexts , 2013, Journal of Cognitive Neuroscience.

[50]  Nelleke Oostdijk,et al.  The Construction of a 500-Million-Word Reference Corpus of Contemporary Written Dutch , 2013, Essential Speech and Language Technology for Dutch.

[51]  Roland Schäfer,et al.  Building Large Corpora from the Web Using a New Efficient Tool Chain , 2012, LREC.

[52]  J. Elman,et al.  Generalized event knowledge activation during online sentence comprehension. , 2012, Journal of memory and language.

[53]  C. Van Petten,et al.  Prediction during language comprehension: benefits, costs, and ERP components. , 2012, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[54]  Mark Johnson,et al.  Using Language Models and Latent Semantic Analysis to Characterise the N400m Neural Response , 2011, ALTA.

[55]  Walter Kintsch,et al.  The Construction of Meaning , 2011, Top. Cogn. Sci..

[56]  A. D. Groot Language and Cognition in Bilinguals and Multilinguals: An Introduction , 2010 .

[57]  J. Elman,et al.  Effects of event knowledge in processing verbal arguments. , 2010, Journal of memory and language.

[58]  Marta Kutas,et al.  CHAPTER 15 A Look around at What Lies Ahead: Prediction and Predictability in Language Processing , 2010 .

[59]  Peter Hagoort,et al.  When Elephants Fly: Differential Sensitivity of Right and Left Inferior Frontal Gyri to Discourse and World Knowledge , 2009, Journal of Cognitive Neuroscience.

[60]  Fred L. Drake,et al.  Python 3 Reference Manual , 2009 .

[61]  Patrick F. Reidy An Introduction to Latent Semantic Analysis , 2009 .

[62]  Hartmut Leuthold,et al.  Eye-movements and ERPs reveal the time course of processing negation and remitting counterfactual worlds , 2008, Brain Research.

[63]  Hartmut Leuthold,et al.  Processing local pragmatic anomalies in fictional contexts: evidence from the N400. , 2008, Psychophysiology.

[64]  Keith Rayner,et al.  Effects of context on eye movements when reading about possible and impossible events. , 2008, Journal of experimental psychology. Learning, memory, and cognition.

[65]  Heather J. Ferguson,et al.  Anomalies in real and counterfactual worlds : An eye-movement investigation , 2008 .

[66]  R. Levy Expectation-based syntactic comprehension , 2008, Cognition.

[67]  Anette Rosenbach,et al.  Animacy and grammatical variation—Findings from English genitive variation , 2008 .

[68]  Peter Hagoort,et al.  Beyond the sentence given , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[69]  Mante S. Nieuwland,et al.  Establishing reference in language comprehension: An electrophysiological perspective , 2007, Brain Research.

[70]  Mante S. Nieuwland,et al.  When Peanuts Fall in Love: N400 Evidence for the Power of Discourse , 2005, Journal of Cognitive Neuroscience.

[71]  H. Kolk,et al.  Accessing world knowledge: evidence from N400 and reaction time priming. , 2005, Brain research. Cognitive brain research.

[72]  Walter Kintsch,et al.  An Overview of Top-Down and Bottom-Up Effects in Comprehension: The CI Perspective , 2005 .

[73]  Rolf A. Zwaan,et al.  Updating situation models. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[74]  Walter Kintsch,et al.  Text Comprehension and Discourse Processing , 2003 .

[75]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[76]  Rolf A. Zwaan,et al.  Retrieval from temporally organized situation models. , 1998, Journal of experimental psychology. Learning, memory, and cognition.

[77]  Rolf A. Zwaan,et al.  Situation models in language comprehension and memory. , 1998, Psychological bulletin.

[78]  Rolf A. Zwaan,et al.  Discourse comprehension. , 1997, Annual review of psychology.

[79]  Rolf A. Zwaan,et al.  The Construction of Situation Models in Narrative Comprehension: An Event-Indexing Model , 1995 .

[80]  Arthur C. Graesser,et al.  Dimensions of situation model construction in narrative comprehension. , 1995 .

[81]  M. Kutas In the company of other words: Electrophysiological evidence for single-word and sentence context effects , 1993 .

[82]  M. Kutas,et al.  An Electrophysiological Probe of Incidental Semantic Association , 1989, Journal of Cognitive Neuroscience.

[83]  P. Holcomb Automatic and attentional processing: An event-related brain potential analysis of semantic priming , 1988, Brain and Language.

[84]  S. T. Dumais,et al.  Using latent semantic analysis to improve access to textual information , 1988, CHI '88.

[85]  W. Kintsch The role of knowledge in discourse comprehension: a construction-integration model. , 1988, Psychological review.

[86]  Marta Kutas,et al.  Tracking the Time Course of Meaning Activation , 1988 .

[87]  M. Rugg The effects of semantic priming and work repetition on event-related potentials. , 1985, Psychophysiology.

[88]  C. C. Wood,et al.  Event-related potentials, lexical decision and semantic priming. , 1985, Electroencephalography and clinical neurophysiology.

[89]  M. Kutas,et al.  Brain potentials during reading reflect word expectancy and semantic association , 1984, Nature.

[90]  W. Kintsch,et al.  Strategies of discourse comprehension , 1983 .

[91]  P. Johnson-Laird,et al.  Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness , 1985 .

[92]  A. Garnham Mental models as representations of text , 1981, Memory & cognition.

[93]  Philip N. Johnson-Laird,et al.  Mental Models in Cognitive Science , 1980, Cogn. Sci..

[94]  I. Fischler,et al.  Automatic and attentional processes in the effects of sentence contexts on word recognition , 1979 .

[95]  Walter Kintsch,et al.  Toward a model of text comprehension and production. , 1978 .

[96]  J. Bransford,et al.  Sentence memory: A constructive versus interpretive approach ☆ ☆☆ , 1972 .