Different kinds of cognitive plausibility: why are transformers better than RNNs at predicting N400 amplitude?

Despite being designed for performance rather than cognitive plausibility, transformer language models have been found to be better at predicting metrics used to assess human language comprehension than language models with other architectures, such as recurrent neural networks. Based on how well they predict the N400, a neural signal associated with processing difficulty, we propose and provide evidence for one possible explanation—their predictions are affected by the preceding context in a way analogous to the effect of semantic facilitation in humans.

[1]  A. Friederici.  Event-related brain potential studies in language , 2004, Current neurology and neuroscience reports.

[2]  B. Mittelman,et al.  On control. , 1979, Dental management.

[3]  C. Petten.  A comparison of lexical and sentence-level context effects in event-related potentials , 1993 .

[4]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[5]  Katherine A. DeLong,et al.  Comprehending surprising sentences: sensitivity of post-N400 positivities to contextual congruity and semantic relatedness , 2020, Language, cognition and neuroscience.

[6]  Wilson L. Taylor,et al.  'Cloze' Readability Scores as Indices of Individual Differences in Comprehension and Aptitude: Erratum. , 1957 .

[7]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[8]  G. Kuperberg,et al.  Word predictability effects are linear, not logarithmic: Implications for probabilistic models of sentence comprehension. , 2021, Journal of memory and language.

[9]  S. Frank,et al.  The ERP response to the amount of information conveyed by words in sentences , 2015, Brain and Language.

[10]  Roger Levy,et al.  On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior , 2020, CogSci.

[11]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[12]  Mark Johnson,et al.  Using Language Models and Latent Semantic Analysis to Characterise the N400m Neural Response , 2011, ALTA.

[13]  Allyson Ettinger,et al.  What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models , 2019, TACL.

[14]  Allyson Ettinger,et al.  Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Priming , 2020, EMNLP.

[15]  Gina R Kuperberg,et al.  Separate streams or probabilistic inference? What the N400 can tell us about the comprehension of events , 2016, Language, cognition and neuroscience.

[16]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[17]  Benjamin K. Bergen,et al.  How well does surprisal explain N400 amplitude under different experimental conditions? , 2020, CONLL.

[18]  Gina R. Kuperberg,et al.  A Tale of Two Positivities (and the N400): Distinct neural signatures are evoked by confirmed and violated predictions at different levels of representation , 2018, bioRxiv.

[19]  Edmund C. Lalor,et al.  Electrophysiological Correlates of Semantic Dissimilarity Reflect the Comprehension of Natural, Narrative Speech , 2017, Current Biology.

[20]  Steven G. Luke,et al.  Limits on lexical prediction during reading , 2016, Cognitive Psychology.

[21]  James L. McClelland,et al.  Modelling the N400 brain potential as change in a probabilistic representation of meaning , 2018, Nature Human Behaviour.

[22]  C. Van Petten,et al.  Prediction during language comprehension: benefits, costs, and ERP components. , 2012, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[23]  Hartmut Fitz,et al.  Getting real about Semantic Illusions: Rethinking the functional role of the P600 in language comprehension , 2012, Brain Research.

[24]  Frank Keller,et al.  Cognitively Plausible Models of Human Language Processing , 2010, ACL.

[25]  Roel M. Willems,et al.  Word predictability and semantic similarity show distinct patterns of brain activity during language comprehension , 2017 .

[26]  C. Van Petten,et al.  Lexical versus conceptual anticipation during sentence processing: frontal positivity and N400 ERP components. , 2012, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[27]  Stefan L. Frank,et al.  Evaluating information-theoretic measures of word prediction in naturalistic sentence reading , 2019, Neuropsychologia.

[28]  Stefan L. Frank,et al.  Comparing Transformers and RNNs on predicting human sentence processing data , 2020, ArXiv.

[29]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[30]  D. Caplan,et al.  Electrophysiological distinctions in processing conceptual relationships within simple sentences. , 2003, Brain research. Cognitive brain research.

[31]  Allyson Ettinger,et al.  Modeling N400 amplitude using vector space models of word representation , 2016, CogSci.

[32]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.

[33]  Yonghui Wu,et al.  Exploring the Limits of Language Modeling , 2016, ArXiv.

[34]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[35]  M. Kutas.  In the company of other words: Electrophysiological evidence for single-word and sentence context effects , 1993 .

[36]  Kara D. Federmeier,et al.  A Rose by Any Other Name: Long-Term Memory Structure and Sentence Processing , 1999 .

[37]  Ellen F. Lau,et al.  Dissociating N400 Effects of Prediction from Association in Single-word Contexts , 2013, Journal of Cognitive Neuroscience.