Languagestructureinthebrain:A fixation-relatedfMRIstudyofsyntactic surprisal in reading

article i nfo How is syntactic analysis implemented by the human brain during language comprehension? The current study combined methods from computational linguistics, eyetracking, and fMRI to address this question. Subjects read passages of text presented as paragraphs while their eye movements were recorded in an MRI scanner. We parsed the text using a probabilistic context-free grammar to isolate syntactic difficulty. Syntactic difficulty was quantified as syntactic surprisal, which is related to the expectedness of a given word's syntactic category given its preceding context. We compared words with high and low syntactic surprisal values that were equated for length, frequency, and lexical surprisal, and used fixation-related (FIRE) fMRItomeasure neural activity asso- ciated with syntactic surprisal for each fixated word. We observed greater neural activity for high than low syn- tacticsurprisalintwo predicted cortical regionspreviously identifiedwith syntax: leftinferior frontalgyrus (IFG)

[1]  R W Cox,et al.  Real‐time 3D image registration for functional MRI , 1999, Magnetic resonance in medicine.

[2]  Antal van den Bosch,et al.  Prediction During Natural Language Comprehension. , 2016, Cerebral cortex.

[3]  J. Gee,et al.  Neural basis for sentence comprehension: Grammatical and short‐term memory components , 2002, Human brain mapping.

[4]  E. Kaan,et al.  The brain circuitry of syntactic comprehension , 2002, Trends in Cognitive Sciences.

[5]  J. Gallant,et al.  Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies , 2011, Current Biology.

[6]  Brian Roark,et al.  Probabilistic Top-Down Parsing and Language Modeling , 2001, CL.

[7]  A. Syrota,et al.  The Cortical Representation of Speech , 1993, Journal of Cognitive Neuroscience.

[8]  Moshe Bar,et al.  Predictions: a universal principle in the operation of the human brain , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[9]  A. Friederici,et al.  Localization of early syntactic processes in frontal and temporal cortical areas: A magnetoencephalographic study , 2000, Human brain mapping.

[10]  A. Friederici,et al.  Auditory Language Comprehension: An Event-Related fMRI Study on the Processing of Syntactic and Lexical Information , 2000, Brain and Language.

[11]  Corinna E. Bonhage,et al.  Combined eye tracking and fMRI reveals neural basis of linguistic predictions during sentence comprehension , 2015, Cortex.

[12]  G. Altmann,et al.  The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye-movements , 2003 .

[13]  N. Dronkers,et al.  Lesion analysis of the brain areas involved in language comprehension , 2004, Cognition.

[14]  Marta Kutas,et al.  Expecting Gender: An Event Related Brain Potential Study on the Role of Grammatical Gender in Comprehending a Line Drawing Within a Written Sentence in Spanish , 2003, Cortex.

[15]  John M. Henderson,et al.  Neural Correlates of Fixation Duration during Real-world Scene Viewing: Evidence from Fixation-related (FIRE) fMRI , 2015, Journal of Cognitive Neuroscience.

[16]  Y. Grodzinsky.  The neurology of syntax: Language use without Broca's area , 2000, Behavioral and Brain Sciences.

[17]  Cathy J. Price,et al.  A review and synthesis of the first 20 years of PET and fMRI studies of heard speech, spoken language and reading , 2012, NeuroImage.

[18]  C. Fiebach,et al.  The role of left inferior frontal and superior temporal cortex in sentence comprehension: localizing syntactic and semantic processes. , 2003, Cerebral cortex.

[19]  A. Bachrach.  Imaging neural correlates of syntactic complexity in a naturalistic context , 2008 .

[20]  Yosef Grodzinsky,et al.  Neuroimaging of syntax and syntactic processing , 2006, Current Opinion in Neurobiology.

[21]  Reinhold Kliegl,et al.  Parsing costs as predictors of reading difficulty: An evaluation using the Potsdam Sentence Corpus , 2008, Journal of Eye Movement Research.

[22]  Colin M. Brown,et al.  Anticipating upcoming words in discourse: evidence from ERPs and reading times. , 2005, Journal of experimental psychology. Learning, memory, and cognition.

[23]  Friedemann Pulvermüller,et al.  Modulation of brain activity by multiple lexical and word form variables in visual word recognition: A parametric fMRI study , 2008, NeuroImage.

[24]  Liina Pylkkänen,et al.  The interplay of composition and concept specificity in the left anterior temporal lobe: An MEG study , 2015, NeuroImage.

[25]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[26]  Nancy Kanwisher,et al.  Syntactic processing in the human brain: What we know, what we don’t know, and a suggestion for how to proceed , 2011, Brain and Language.

[27]  John A. Hawkins,et al.  A Performance Theory of Order and Constituency , 1995 .

[28]  Yosef Grodzinsky,et al.  Working memory and syntax interact in Broca's area , 2007, NeuroImage.

[29]  Yosef Grodzinsky,et al.  Neural correlates of syntactic movement: converging evidence from two fMRI experiments , 2004, NeuroImage.

[30]  Jack L. Gallant,et al.  Natural Scene Statistics Account for the Representation of Scene Categories in Human Visual Cortex , 2013, Neuron.

[31]  Floris P. de Lange,et al.  How Prediction Errors Shape Perception, Attention, and Motivation , 2012, Front. Psychology.

[32]  Colin Humphries,et al.  Syntactic and Semantic Modulation of Neural Activity during Auditory Sentence Comprehension , 2006, Journal of Cognitive Neuroscience.

[33]  D. Swinney,et al.  Response of anterior temporal cortex to syntactic and prosodic manipulations during sentence processing , 2005, Human brain mapping.

[34]  Falk Huettig,et al.  Prediction during language processing is a piece of cake--but only for skilled producers. , 2012, Journal of experimental psychology. Human perception and performance.

[35]  M. Pickering,et al.  Toward a mechanistic psychology of dialogue , 2004, Behavioral and Brain Sciences.

[36]  N. Alpert,et al.  Localization of Syntactic Comprehension by Positron Emission Tomography , 1996, Brain and Language.

[37]  M. Pickering,et al.  An integrated theory of language production and comprehension. , 2013, The Behavioral and brain sciences.

[38]  J. Henderson,et al.  Neural correlates of active vision: An fMRI comparison of natural reading and scene viewing , 2015, Neuropsychologia.

[39]  Y. Miyashita,et al.  A syntactic specialization for Broca's area. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[40]  Rutvik H. Desai,et al.  The neural substrates of natural reading: a comparison of normal and nonword text using eyetracking and fMRI , 2014, Front. Hum. Neurosci..

[41]  N. Dronkers.  A new brain region for coordinating speech articulation , 1996, Nature.

[42]  Morten H. Christiansen,et al.  Sequential Expectations: The Role of Prediction-Based Learning in Language , 2010, Top. Cogn. Sci..

[43]  C. J. Price,et al.  An fMRI Study of Syntactic Adaptation , 2004, Journal of Cognitive Neuroscience.

[44]  R W Cox,et al.  AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. , 1996, Computers and biomedical research, an international journal.

[45]  D. Perani,et al.  Syntax and the Brain: Disentangling Grammar by Selective Anomalies , 2001, NeuroImage.

[46]  M. Kronbichler,et al.  Fixation-Related fMRI Analysis in the Domain of Reading Research: Using Self-Paced Eye Movements as Markers for Hemodynamic Brain Responses During Visual Letter String Processing , 2013, Cerebral cortex.

[47]  Kara D. Federmeier,et al.  Finding the right word: Hemispheric asymmetries in the use of sentence context information , 2007, Neuropsychologia.

[48]  John Hale,et al.  A Probabilistic Earley Parser as a Psycholinguistic Model , 2001, NAACL.

[49]  G. Curio,et al.  The Human Thalamus Processes Syntactic and Semantic Language Violations , 2008, Neuron.

[50]  Karl J. Friston.  The free-energy principle: a unified brain theory? , 2010, Nature Reviews Neuroscience.

[51]  John Hale,et al.  Modeling fMRI time courses with linguistic structure at various grain sizes , 2015, CMCL@NAACL-HLT.

[52]  A. Nobre,et al.  The Response of Left Temporal Cortex to Sentences , 2002, Journal of Cognitive Neuroscience.

[53]  Frank Keller,et al.  Data from eye-tracking corpora as evidence for theories of syntactic processing complexity , 2008, Cognition.

[54]  R. Malach,et al.  Intersubject Synchronization of Cortical Activity During Natural Vision , 2004, Science.

[55]  S. Frank,et al.  The ERP response to the amount of information conveyed by words in sentences , 2015, Brain and Language.

[56]  Brian Roark,et al.  Deriving lexical and syntactic expectation-based measures for psycholinguistic modeling via incremental top-down parsing , 2009, EMNLP.

[57]  A. Caramazza,et al.  Dissociation of algorithmic and heuristic processes in language comprehension: Evidence from aphasia , 1976, Brain and Language.

[58]  A. Clark,et al.  Words and the World , 2015 .

[59]  David Caplan,et al.  Task-dependent and task-independent neurovascular responses to syntactic processing , 2008, Cortex.

[60]  Frans W Cornelissen,et al.  Fixation based event‐related fmri analysis: Using eye fixations as events in functional magnetic resonance imaging to reveal cortical processing during the free exploration of visual images , 2012, Human Brain Mapping.

[61]  Ina Bornkessel-Schlesewsky,et al.  Word order and Broca’s region: Evidence for a supra-syntactic perspective , 2009, Brain and Language.

[62]  J. Henderson,et al.  Effects of foveal processing difficulty on the perceptual span in reading: implications for attention and eye movement control. , 1990, Journal of experimental psychology. Learning, memory, and cognition.

[63]  E. Gibson.  The dependency locality theory: A distance-based theory of linguistic complexity. , 2000 .

[64]  Nathaniel J. Smith,et al.  The effect of word predictability on reading time is logarithmic , 2013, Cognition.

[65]  R. Malach,et al.  Syntactic structure building in the anterior temporal lobe during natural story listening , 2012, Brain and Language.

[66]  Frank Keller,et al.  Incremental, Predictive Parsing with Psycholinguistically Motivated Tree-Adjoining Grammar , 2013, CL.

[67]  G. Altmann,et al.  Incremental interpretation at verbs: restricting the domain of subsequent reference , 1999, Cognition.

[68]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[69]  Matthias Schlesewsky,et al.  Processing linguistic complexity and grammaticality in the left frontal cortex. , 2005, Cerebral cortex.

[70]  E. Gibson.  Linguistic complexity: locality of syntactic dependencies , 1998, Cognition.

[71]  Philip Lieberman,et al.  Selective speech motor, syntax and cognitive deficits associated with bilateral damage to the putamen and the head of the caudate nucleus: a case study , 1998, Neuropsychologia.

[72]  William D. Marslen-Wilson,et al.  Left inferior frontal cortex and syntax: function, structure and behaviour in patients with left hemisphere damage , 2011, Brain : a journal of neurology.

[73]  Sylvia Vitello,et al.  Localising semantic and syntactic processing in spoken and written language comprehension: An Activation Likelihood Estimation meta-analysis , 2015, Brain and Language.

[74]  Michael S. Beauchamp,et al.  A new method for improving functional-to-structural MRI alignment using local Pearson correlation , 2009, NeuroImage.

[75]  Marc Brysbaert,et al.  Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English , 2009, Behavior research methods.

[76]  R. Levy.  Expectation-based syntactic comprehension , 2008, Cognition.

[77]  Istvan Molnar-Szakacs,et al.  Beyond superior temporal cortex: intersubject correlations in narrative speech comprehension. , 2008, Cerebral cortex.

[78]  P. Lieberman.  Human Language and Our Reptilian Brain: The Subcortical Bases of Speech, Syntax, and Thought , 2001, Perspectives in biology and medicine.

[79]  M. Torrens.  Co-Planar Stereotaxic Atlas of the Human Brain—3-Dimensional Proportional System: An Approach to Cerebral Imaging, J. Talairach, P. Tournoux. Georg Thieme Verlag, New York (1988), 122 pp., 130 figs. DM 268 , 1990 .

[80]  Adrian Staub,et al.  The Effect of Lexical Predictability on Eye Movements in Reading: Critical Review and Theoretical Interpretation , 2015, Lang. Linguistics Compass.

[81]  Steven L. Small,et al.  Repetition Suppression for Spoken Sentences and the Effect of Task Demands , 2006, Journal of Cognitive Neuroscience.

[82]  Arthur M Jacobs,et al.  Fact vs fiction--how paratextual information shapes our reading processes. , 2014, Social cognitive and affective neuroscience.

[83]  Gregory Hickok,et al.  Selective attention to semantic and syntactic features modulates sentence processing networks in anterior temporal cortex. , 2009, Cerebral cortex.

[84]  Ellen F. Lau,et al.  Spatiotemporal Signatures of Lexical-Semantic Prediction. , 2016, Cerebral cortex.

[85]  M. Just,et al.  Brain Activation Modulated by Sentence Comprehension , 1996, Science.

[86]  Erik D. Reichle,et al.  The E-Z Reader model of eye-movement control in reading: Comparisons to other models , 2003, Behavioral and Brain Sciences.

[87]  S. Bookheimer,et al.  Form and Content Dissociating Syntax and Semantics in Sentence Comprehension , 1999, Neuron.

[88]  Weijia Ni,et al.  Sentence complexity and input modality effects in sentence comprehension: an fMRI study , 2004, NeuroImage.

[89]  Marta Kutas,et al.  CHAPTER 15 A Look around at What Lies Ahead: Prediction and Predictability in Language Processing , 2010 .

[90]  A. Borst.  Seeing smells: imaging olfactory learning in bees , 1999, Nature Neuroscience.

[91]  Gina R Kuperberg,et al.  What do we mean by prediction in language comprehension? , 2016, Language, cognition and neuroscience.

[92]  Doris-Eva Bamiou,et al.  The insula (Island of Reil) and its role in auditory processing Literature review , 2003, Brain Research Reviews.

[93]  Erik D. Reichle,et al.  Eye movements in reading and information processing: Keith Rayner’s 40 year legacy , 2016 .

[94]  John M. Henderson,et al.  Neural correlates of fixation duration in natural reading: Evidence from fixation-related fMRI , 2015, NeuroImage.

[95]  J. Trueswell,et al.  Parametric effects of syntactic–semantic conflict in Broca’s area during sentence processing , 2012, Brain and Language.

[96]  A. Friederici,et al.  The brain basis of syntactic processes: functional imaging and lesion studies , 2003, NeuroImage.

[97]  S. Zeki,et al.  Functional brain mapping during free viewing of natural scenes , 2004, Human brain mapping.

[98]  T. Hendler,et al.  The Neural Reality of Syntactic Transformations , 2003, Psychological science.

[99]  G. Waters,et al.  Activation of Broca's area by syntactic processing under conditions of concurrent articulation , 2000, Human brain mapping.

[100]  G. McCarthy,et al.  Echo-planar magnetic resonance imaging studies of frontal cortex activation during word generation in humans. , 1993, Proceedings of the National Academy of Sciences of the United States of America.

[101]  William W. Graves,et al.  Neural Systems for Reading Aloud: A Multiparametric Approach , 2009, Cerebral cortex.