Understanding language-elicited EEG data by predicting it from a fine-tuned language model

Electroencephalography (EEG) recordings of brain activity taken while participants read or listen to language are widely used within the cognitive neuroscience and psycholinguistics communities as a tool to study language comprehension. Several time-locked stereotyped EEG responses to word-presentations – known collectively as event-related potentials (ERPs) – are thought to be markers for semantic or syntactic processes that take place during comprehension. However, the characterization of each individual ERP in terms of what features of a stream of language trigger the response remains controversial. Improving this characterization would make ERPs a more useful tool for studying language comprehension. We take a step towards better understanding the ERPs by finetuning a language model to predict them. This new approach to analysis shows for the first time that all of the ERPs are predictable from embeddings of a stream of language. Prior work has only found two of the ERPs to be predictable. In addition to this analysis, we examine which ERPs benefit from sharing parameters during joint training. We find that two pairs of ERPs previously identified in the literature as being related to each other benefit from joint training, while several other pairs of ERPs that benefit from joint training are suggestive of potential relationships. Extensions of this analysis that further examine what kinds of information in the model embeddings relate to each ERP have the potential to elucidate the processes involved in human language comprehension.

[1]  Sebastian Ruder,et al.  Universal Language Model Fine-tuning for Text Classification , 2018, ACL.

[2]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[3]  D. Kemmerer.  Cognitive Neuroscience of Language , 2014 .

[4]  M. Kutas,et al.  Who Did What and When? Using Word- and Clause-Level ERPs to Monitor Working Memory Usage in Reading , 1995, Journal of Cognitive Neuroscience.

[5]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[6]  A. Friederici,et al.  Differential task effects on semantic and syntactic processes as revealed by ERPs. , 2002, Brain research. Cognitive brain research.

[7]  G. Mulder,et al.  When syntax meets semantics. , 1997, Psychophysiology.

[8]  K. Rayner,et al.  Eye-Movement Control in Reading , 2006 .

[9]  Tom M. Mitchell,et al.  Aligning context-based statistical models of language with brain activity during reading , 2014, EMNLP.

[10]  M. Kutas,et al.  Expect the Unexpected: Event-related Brain Response to Morphosyntactic Violations , 1998 .

[11]  A. Friederici,et al.  Electrophysiological Evidence for Two Steps in Syntactic Analysis: Early Automatic and Late Controlled Processes , 1999, Journal of Cognitive Neuroscience.

[12]  Emily J. Mayberry,et al.  Coherent concepts are computed in the anterior temporal lobes , 2010, Proceedings of the National Academy of Sciences.

[13]  Richard Socher,et al.  Pointer Sentinel Mixture Models , 2016, ICLR.

[14]  C. Van Petten,et al.  Prediction during language comprehension: benefits, costs, and ERP components. , 2012, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[15]  N. Draper,et al.  An Alternative Family of Transformations , 1980 .

[16]  Gina R. Kuperberg,et al.  Neural mechanisms of language comprehension: Challenges to syntax , 2007, Brain Research.

[17]  D. Caplan,et al.  Electrophysiological distinctions in processing conceptual relationships within simple sentences. , 2003, Brain research. Cognitive brain research.

[18]  S. Frank,et al.  The ERP response to the amount of information conveyed by words in sentences , 2015, Brain and Language.

[19]  Robin L Thompson,et al.  Reading time data for evaluating broad-coverage models of English sentence processing , 2013, Behavior research methods.

[20]  K. Rayner.  The 35th Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search , 2009, Quarterly journal of experimental psychology.

[21]  Falk Huettig,et al.  Four central questions about prediction in language processing , 2015, Brain Research.

[22]  M. Kutas,et al.  Psycholinguistics Electrified II (1994–2005) , 2006 .

[23]  Peter Hagoort,et al.  Real-time semantic compensation in patients with agrammatic comprehension: Electrophysiological evidence for multiple-route plasticity , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[24]  John Hale,et al.  Finding syntax in human encephalography with beam search , 2018, ACL.

[25]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[26]  D. Poeppel,et al.  The cortical organization of speech processing , 2007, Nature Reviews Neuroscience.

[27]  Markus Jokela,et al.  Event-related potentials suggest early interaction between syntax and semantics during on-line sentence comprehension , 2005, Neuroscience Letters.

[28]  A. Friederici.  The brain basis of language processing: from structure to function. , 2011, Physiological reviews.

[29]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.

[30]  Richard Socher,et al.  Regularizing and Optimizing LSTM Language Models , 2017, ICLR.

[31]  Sebastian Ruder,et al.  Fine-tuned Language Models for Text Classification , 2018, ArXiv.

[32]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[33]  Colin M. Brown,et al.  Syntax-related ERP-effects in Dutch. , 2003, Brain research. Cognitive brain research.