Language structure in the brain: A fixation-related fMRI study of syntactic surprisal in reading

How is syntactic analysis implemented by the human brain during language comprehension? The current study combined methods from computational linguistics, eyetracking, and fMRI to address this question. Subjects read passages of text presented as paragraphs while their eye movements were recorded in an MRI scanner. We parsed the text using a probabilistic context-free grammar to isolate syntactic difficulty. Syntactic difficulty was quantified as syntactic surprisal, which is related to the expectedness of a given word's syntactic category given its preceding context. We compared words with high and low syntactic surprisal values that were equated for length, frequency, and lexical surprisal, and used fixation-related (FIRE) fMRI to measure neural activity associated with syntactic surprisal for each fixated word. We observed greater neural activity for high than low syntactic surprisal in two predicted cortical regions previously identified with syntax: left inferior frontal gyrus (IFG) and less robustly, left anterior superior temporal lobe (ATL). These results support the hypothesis that left IFG and ATL play a central role in syntactic analysis during language comprehension. More generally, the results suggest a broader cortical network associated with syntactic prediction that includes increased activity in bilateral IFG and insula, as well as fusiform and right lingual gyri.

[1]  T. Hendler,et al.  The Neural Reality of Syntactic Transformations , 2003, Psychological science.

[2]  William D. Marslen-Wilson,et al.  Left inferior frontal cortex and syntax: function, structure and behaviour in patients with left hemisphere damage , 2011, Brain : a journal of neurology.

[3]  Sylvia Vitello,et al.  Localising semantic and syntactic processing in spoken and written language comprehension: An Activation Likelihood Estimation meta-analysis , 2015, Brain and Language.

[4]  G. Waters,et al.  Activation of Broca's area by syntactic processing under conditions of concurrent articulation , 2000, Human brain mapping.

[5]  Doris-Eva Bamiou,et al.  The insula (Island of Reil) and its role in auditory processing Literature review , 2003, Brain Research Reviews.

[6]  C. J. Price,et al.  An fMRI Study of Syntactic Adaptation , 2004, Journal of Cognitive Neuroscience.

[7]  Cathy J. Price,et al.  A review and synthesis of the first 20 years of PET and fMRI studies of heard speech, spoken language and reading , 2012, NeuroImage.

[8]  G. McCarthy,et al.  Echo-planar magnetic resonance imaging studies of frontal cortex activation during word generation in humans. , 1993, Proceedings of the National Academy of Sciences of the United States of America.

[9]  M. Pickering,et al.  Toward a mechanistic psychology of dialogue , 2004, Behavioral and Brain Sciences.

[10]  John A. Hawkins,et al.  A Performance Theory of Order and Constituency , 1995 .

[11]  Yosef Grodzinsky,et al.  Neural correlates of syntactic movement: converging evidence from two fMRI experiments , 2004, NeuroImage.

[12]  Michael S. Beauchamp,et al.  A new method for improving functional-to-structural MRI alignment using local Pearson correlation , 2009, NeuroImage.

[13]  R. Levy Expectation-based syntactic comprehension , 2008, Cognition.

[14]  Marc Brysbaert,et al.  Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English , 2009, Behavior research methods.

[15]  Erik D. Reichle,et al.  Eye movements in reading and information processing: Keith Rayner’s 40 year legacy , 2016 .

[16]  John Hale,et al.  A Probabilistic Earley Parser as a Psycholinguistic Model , 2001, NAACL.

[17]  Istvan Molnar-Szakacs,et al.  Beyond superior temporal cortex: intersubject correlations in narrative speech comprehension. , 2008, Cerebral cortex.

[18]  C. Fiebach,et al.  The role of left inferior frontal and superior temporal cortex in sentence comprehension: localizing syntactic and semantic processes. , 2003, Cerebral cortex.

[19]  William W. Graves,et al.  Neural Systems for Reading Aloud: A Multiparametric Approach , 2009, Cerebral cortex.

[20]  N. Alpert,et al.  Localization of Syntactic Comprehension by Positron Emission Tomography , 1996, Brain and Language.

[21]  M. Pickering,et al.  An integrated theory of language production and comprehension. , 2013, The Behavioral and brain sciences.

[22]  R W Cox,et al.  Real‐time 3D image registration for functional MRI , 1999, Magnetic resonance in medicine.

[23]  N. Dronkers A new brain region for coordinating speech articulation , 1996, Nature.

[24]  Karl J. Friston The free-energy principle: a unified brain theory? , 2010, Nature Reviews Neuroscience.

[25]  Antal van den Bosch,et al.  Prediction During Natural Language Comprehension. , 2016, Cerebral cortex.

[26]  A. Nobre,et al.  The Response of Left Temporal Cortex to Sentences , 2002, Journal of Cognitive Neuroscience.

[27]  R W Cox,et al.  AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. , 1996, Computers and biomedical research, an international journal.

[28]  R. Malach,et al.  Intersubject Synchronization of Cortical Activity During Natural Vision , 2004, Science.

[29]  A. Friederici,et al.  Auditory Language Comprehension: An Event-Related fMRI Study on the Processing of Syntactic and Lexical Information , 2000, Brain and Language.

[30]  G. Altmann,et al.  The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye-movements , 2003 .

[31]  Nathaniel J. Smith,et al.  The effect of word predictability on reading time is logarithmic , 2013, Cognition.

[32]  S. Frank,et al.  The ERP response to the amount of information conveyed by words in sentences , 2015, Brain and Language.

[33]  Brian Roark,et al.  Deriving lexical and syntactic expectation-based measures for psycholinguistic modeling via incremental top-down parsing , 2009, EMNLP.

[34]  A. Clark Whatever next? Predictive brains, situated agents, and the future of cognitive science. , 2013, The Behavioral and brain sciences.

[35]  A. Friederici,et al.  Localization of early syntactic processes in frontal and temporal cortical areas: A magnetoencephalographic study , 2000, Human brain mapping.

[36]  N. Dronkers,et al.  Lesion analysis of the brain areas involved in language comprehension , 2004, Cognition.

[37]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[38]  Nancy Kanwisher,et al.  Syntactic processing in the human brain: What we know, what we don’t know, and a suggestion for how to proceed , 2011, Brain and Language.

[39]  R. Malach,et al.  Syntactic structure building in the anterior temporal lobe during natural story listening , 2012, Brain and Language.

[40]  Corinna E. Bonhage,et al.  Combined eye tracking and fMRI reveals neural basis of linguistic predictions during sentence comprehension , 2015, Cortex.

[41]  Yosef Grodzinsky,et al.  Neuroimaging of syntax and syntactic processing , 2006, Current Opinion in Neurobiology.

[42]  John M. Henderson,et al.  Neural Correlates of Fixation Duration during Real-world Scene Viewing: Evidence from Fixation-related (FIRE) fMRI , 2015, Journal of Cognitive Neuroscience.

[43]  A. Bachrach Imaging neural correlates of syntactic complexity in a naturalistic context , 2008 .

[44]  Colin M. Brown,et al.  Anticipating upcoming words in discourse: evidence from ERPs and reading times. , 2005, Journal of experimental psychology. Learning, memory, and cognition.

[45]  Friedemann Pulvermüller,et al.  Modulation of brain activity by multiple lexical and word form variables in visual word recognition: A parametric fMRI study , 2008, NeuroImage.

[46]  Liina Pylkkänen,et al.  The interplay of composition and concept specificity in the left anterior temporal lobe: An MEG study , 2015, NeuroImage.

[47]  Jack L. Gallant,et al.  Natural Scene Statistics Account for the Representation of Scene Categories in Human Visual Cortex , 2013, Neuron.

[48]  Adrian Staub,et al.  The Effect of Lexical Predictability on Eye Movements in Reading: Critical Review and Theoretical Interpretation , 2015, Lang. Linguistics Compass.

[49]  John M. Henderson,et al.  Neural correlates of fixation duration in natural reading: Evidence from fixation-related fMRI , 2015, NeuroImage.

[50]  J. Trueswell,et al.  Parametric effects of syntactic–semantic conflict in Broca’s area during sentence processing , 2012, Brain and Language.

[51]  A. Friederici,et al.  The brain basis of syntactic processes: functional imaging and lesion studies , 2003, NeuroImage.

[52]  J. Gee,et al.  Neural basis for sentence comprehension: Grammatical and short‐term memory components , 2002, Human brain mapping.

[53]  J. Gallant,et al.  Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies , 2011, Current Biology.

[54]  Falk Huettig,et al.  Prediction during language processing is a piece of cake--but only for skilled producers. , 2012, Journal of experimental psychology. Human perception and performance.

[55]  Moshe Bar,et al.  Predictions: a universal principle in the operation of the human brain , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[56]  Ina Bornkessel-Schlesewsky,et al.  Word order and Broca’s region: Evidence for a supra-syntactic perspective , 2009, Brain and Language.

[57]  Kara D. Federmeier,et al.  Finding the right word: Hemispheric asymmetries in the use of sentence context information , 2007, Neuropsychologia.

[58]  J. Henderson,et al.  Neural correlates of active vision: An fMRI comparison of natural reading and scene viewing , 2015, Neuropsychologia.

[59]  S. Zeki,et al.  Functional brain mapping during free viewing of natural scenes , 2004, Human brain mapping.

[60]  E. Gibson The dependency locality theory: A distance-based theory of linguistic complexity. , 2000 .

[61]  Marta Kutas,et al.  Expecting Gender: An Event Related Brain Potential Study on the Role of Grammatical Gender in Comprehending a Line Drawing Within a Written Sentence in Spanish , 2003, Cortex.

[62]  Frank Keller,et al.  Incremental, Predictive Parsing with Psycholinguistically Motivated Tree-Adjoining Grammar , 2013, CL.

[63]  Yosef Grodzinsky,et al.  Working memory and syntax interact in Broca's area , 2007, NeuroImage.

[64]  Floris P. de Lange,et al.  How Prediction Errors Shape Perception, Attention, and Motivation , 2012, Front. Psychology.

[65]  David R. Dowty,et al.  Natural Language Parsing: Psychological, Computational, and Theoretical Perspectives , 1985 .

[66]  G. Altmann,et al.  Incremental interpretation at verbs: restricting the domain of subsequent reference , 1999, Cognition.

[67]  E. Kaan,et al.  The brain circuitry of syntactic comprehension , 2002, Trends in Cognitive Sciences.

[68]  Brian Roark,et al.  Probabilistic Top-Down Parsing and Language Modeling , 2001, CL.

[69]  A. Syrota,et al.  The Cortical Representation of Speech , 1993, Journal of Cognitive Neuroscience.

[70]  Reinhold Kliegl,et al.  Parsing costs as predictors of reading difficulty: An evaluation using the Potsdam Sentence Corpus , 2008, Journal of Eye Movement Research.

[71]  Y. Miyashita,et al.  A syntactic specialization for Broca's area. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[72]  Rutvik H. Desai,et al.  The neural substrates of natural reading: a comparison of normal and nonword text using eyetracking and fMRI , 2014, Front. Hum. Neurosci..

[73]  Y. Grodzinsky The neurology of syntax: Language use without Broca's area , 2000, Behavioral and Brain Sciences.

[74]  M. Bar Predictions in the brain : using our past to generate a future , 2011 .

[75]  David Caplan,et al.  Task-dependent and task-independent neurovascular responses to syntactic processing , 2008, Cortex.

[76]  Frans W Cornelissen,et al.  Fixation based event‐related fmri analysis: Using eye fixations as events in functional magnetic resonance imaging to reveal cortical processing during the free exploration of visual images , 2012, Human Brain Mapping.

[77]  J. Henderson,et al.  Effects of foveal processing difficulty on the perceptual span in reading: implications for attention and eye movement control. , 1990, Journal of experimental psychology. Learning, memory, and cognition.

[78]  B. Velichkovsky,et al.  Eye typing in application: A comparison of two interfacing systems with ALS patients , 2008 .

[80]  M. Just,et al.  Brain Activation Modulated by Sentence Comprehension , 1996, Science.

[81]  Erik D. Reichle,et al.  The E-Z Reader model of eye-movement control in reading: Comparisons to other models , 2003, Behavioral and Brain Sciences.

[82]  S. Bookheimer,et al.  Form and Content Dissociating Syntax and Semantics in Sentence Comprehension , 1999, Neuron.

[83]  Weijia Ni,et al.  Sentence complexity and input modality effects in sentence comprehension: an fMRI study , 2004, NeuroImage.

[84]  Marta Kutas,et al.  CHAPTER 15 A Look around at What Lies Ahead: Prediction and Predictability in Language Processing , 2010 .

[85]  Steven L. Small,et al.  Repetition Suppression for Spoken Sentences and the Effect of Task Demands , 2006, Journal of Cognitive Neuroscience.

[86]  Arthur M Jacobs,et al.  Fact vs fiction--how paratextual information shapes our reading processes. , 2014, Social cognitive and affective neuroscience.

[87]  Gregory Hickok,et al.  Selective attention to semantic and syntactic features modulates sentence processing networks in anterior temporal cortex. , 2009, Cerebral cortex.

[88]  Ellen F. Lau,et al.  Spatiotemporal Signatures of Lexical-Semantic Prediction. , 2016, Cerebral cortex.

[89]  K. Rayner Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[90]  Colin Humphries,et al.  Syntactic and Semantic Modulation of Neural Activity during Auditory Sentence Comprehension , 2006, Journal of Cognitive Neuroscience.

[91]  D. Swinney,et al.  Response of anterior temporal cortex to syntactic and prosodic manipulations during sentence processing , 2005, Human brain mapping.

[92]  D. Perani,et al.  Syntax and the Brain: Disentangling Grammar by Selective Anomalies , 2001, NeuroImage.

[93]  M. Kronbichler,et al.  Fixation-Related fMRI Analysis in the Domain of Reading Research: Using Self-Paced Eye Movements as Markers for Hemodynamic Brain Responses During Visual Letter String Processing , 2013, Cerebral cortex.

[94]  Matthias Schlesewsky,et al.  Processing linguistic complexity and grammaticality in the left frontal cortex. , 2005, Cerebral cortex.

[95]  E. Gibson Linguistic complexity: locality of syntactic dependencies , 1998, Cognition.

[96]  Philip Lieberman,et al.  Selective speech motor, syntax and cognitive deficits associated with bilateral damage to the putamen and the head of the caudate nucleus: a case study , 1998, Neuropsychologia.

[97]  A. Caramazza,et al.  Dissociation of algorithmic and heuristic processes in language comprehension: Evidence from aphasia , 1976, Brain and Language.

[98]  A. Clark,et al.  Words and the World , 2015 .

[99]  A. Borst Seeing smells: imaging olfactory learning in bees , 1999, Nature Neuroscience.

[100]  Gina R Kuperberg,et al.  What do we mean by prediction in language comprehension? , 2016, Language, cognition and neuroscience.

[101]  G. Curio,et al.  The Human Thalamus Processes Syntactic and Semantic Language Violations , 2008, Neuron.

[102]  J. Stevenson,et al.  Human language and our reptilian brain: The subcortical bases of speech, syntax, and thought , 2001 .

[103]  John Hale,et al.  Modeling fMRI time courses with linguistic structure at various grain sizes , 2015, CMCL@NAACL-HLT.

[104]  Frank Keller,et al.  Data from eye-tracking corpora as evidence for theories of syntactic processing complexity , 2008, Cognition.

[105]  Morten H. Christiansen,et al.  Sequential Expectations: The Role of Prediction-Based Learning in Language , 2010, Top. Cogn. Sci..