The Cortical Organization of Syntactic Processing Is Supramodal: Evidence from American Sign Language

Abstract Areas within the left-lateralized neural network for language have been found to be sensitive to syntactic complexity in spoken and written language. Previous research has revealed that these areas are active for sign language as well, but whether these areas are specifically responsive to syntactic complexity in sign language independent of lexical processing has yet to be found. To investigate the question, we used fMRI to neuroimage deaf native signers' comprehension of 180 sign strings in American Sign Language (ASL) with a picture-probe recognition task. The ASL strings were all six signs in length but varied at three levels of syntactic complexity: sign lists, two-word sentences, and complex sentences. Syntactic complexity significantly affected comprehension and memory, both behaviorally and neurally, by facilitating accuracy and response time on the picture-probe recognition task and eliciting a left lateralized activation response pattern in anterior and posterior superior temporal sulcus (aSTS and pSTS). Minimal or absent syntactic structure reduced picture-probe recognition and elicited activation in bilateral pSTS and occipital-temporal cortex. These results provide evidence from a sign language, ASL, that the combinatorial processing of anterior STS and pSTS is supramodal in nature. The results further suggest that the neurolinguistic processing of ASL is characterized by overlapping and separable neural systems for syntactic and lexical processing.

[1]  Peter Hagoort,et al.  The neurobiology of language beyond single words. , 2014, Annual review of neuroscience.

[2]  Judy Reilly,et al.  The MacArthur Communicative Development Inventory: Normative Data for American Sign Language. , 2002, Journal of deaf studies and deaf education.

[3]  A. Senghas,et al.  Children Creating Core Properties of Language: Evidence from an Emerging Sign Language in Nicaragua , 2004, Science.

[4]  G. Hickok,et al.  The role of the left frontal operculum in sign language aphasia , 1996 .

[5]  E. Halgren,et al.  Neurolinguistic processing when the brain matures without language , 2017, Cortex.

[6]  R. Mayberry,et al.  Age of acquisition effects on the functional organization of language in the adult brain , 2011, Brain and Language.

[7]  Daphne Bavelier,et al.  Neural systems supporting linguistic structure, linguistic experience, and symbolic communication in sign language and gesture , 2015, Proceedings of the National Academy of Sciences.

[8]  Ellen F. Lau,et al.  The role of the IFG and pSTS in syntactic prediction: Evidence from a parametric study of hierarchical structure in fMRI , 2017, Cortex.

[9]  Alfred Anwander,et al.  Segregating the core computational faculty of human language from working memory , 2009, Proceedings of the National Academy of Sciences.

[10]  Antonio Moreno,et al.  Brain correlates of constituent structure in sign language comprehension , 2018, NeuroImage.

[11]  N. Kanwisher,et al.  New method for fMRI investigations of language: defining ROIs functionally in individual subjects. , 2010, Journal of neurophysiology.

[12]  Joel R. Meyer,et al.  Modality independence of word comprehension , 2002, Human brain mapping.

[13]  Paul A. Taylor,et al.  FMRI Clustering in AFNI: False-Positive Rates Redux , 2017, Brain Connect..

[14]  Karin Harbusch,et al.  Supramodal Sentence Processing in the Human Brain: fMRI Evidence for the Influence of Syntactic Complexity in More Than 200 Participants , 2019, bioRxiv.

[15]  Bernard Mazoyer,et al.  Impact of modality and linguistic complexity during reading and listening tasks , 2007, NeuroImage.

[16]  Valerie A. Carr,et al.  Spatiotemporal Dynamics of Modality-Specific and Supramodal Word Processing , 2003, Neuron.

[17]  James R. Booth,et al.  Functional Anatomy of Intra- and Cross-Modal Lexical Tasks , 2002, NeuroImage.

[18]  Y. Grodzinsky,et al.  The battle for Broca’s region , 2008, Trends in Cognitive Sciences.

[19]  Angela D. Friederici,et al.  The language skeleton after dissecting meaning: A functional segregation within Broca's Area , 2015, NeuroImage.

[20]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[21]  Nancy Kanwisher,et al.  Neural correlate of the construction of sentence meaning , 2016, Proceedings of the National Academy of Sciences.

[22]  Jeffrey L. Elman,et al.  Spatiotemporal Dynamics of Bilingual Word Processing , 2022 .

[23]  Karen Emmorey,et al.  Shared neural correlates for building phrases in signed and spoken language , 2018, Scientific Reports.

[24]  J. Kegl.  Language emergence in a language-ready brain: Acquisition , 2002 .

[25]  Aaron J. Newman,et al.  Brain systems mediating semantic and syntactic processing in deaf native signers: Biological invariance and modality specificity , 2009, Proceedings of the National Academy of Sciences.

[26]  Meghan Zvaigzne,et al.  Subjective frequency ratings for 432 ASL signs , 2014, Behavior research methods.

[27]  M. Sereno,et al.  Neural organization for recognition of grammatical and emotional facial expressions in deaf ASL signers and hearing nonsigners. , 2005, Brain research. Cognitive brain research.

[28]  Eric Halgren,et al.  Neural language processing in adolescent first-language learners. , 2014, Cerebral cortex.

[29]  A. A. Wijers,et al.  Localizing components of a complex task: sentence processing and working memory , 1998, Neuroreport.

[30]  John C. Trueswell,et al.  Co-localization of Stroop and Syntactic Ambiguity Resolution in Broca's Area: Implications for the Neural Basis of Sentence Processing , 2009, Journal of Cognitive Neuroscience.

[31]  A. Nobre,et al.  The Response of Left Temporal Cortex to Sentences , 2002, Journal of Cognitive Neuroscience.

[32]  A. Syrota,et al.  The Cortical Representation of Speech , 1993, Journal of Cognitive Neuroscience.

[33]  Stephen M. Wilson,et al.  Convergence of spoken and written language processing in the superior temporal sulcus , 2018, NeuroImage.

[34]  R. C. Oldfield.  The assessment and analysis of handedness: the Edinburgh inventory. , 1971, Neuropsychologia.

[35]  J. Talairach,et al.  Co-Planar Stereotaxic Atlas of the Human Brain: 3-Dimensional Proportional System: An Approach to Cerebral Imaging , 1988 .

[36]  William Matchin,et al.  A neuronal retuning hypothesis of sentence-specificity in Broca’s area , 2017, Psychonomic Bulletin & Review.

[37]  Denis Cousineau,et al.  Confidence intervals in within-subject designs: A simpler solution to Loftus and Masson's method , 2005 .

[38]  R. Kluender,et al.  Rethinking the critical period for language: New insights into an old question from American Sign Language. , 2018, Bilingualism.

[39]  William Matchin,et al.  Broca's Area, Sentence Comprehension, and Working Memory: An fMRI Study , 2008, Frontiers in human neuroscience.

[40]  P. Hagoort.  Nodes and networks in the neural architecture for language: Broca's region and beyond , 2014, Current Opinion in Neurobiology.

[41]  F. Fabbro,et al.  Language Brain Representation in Bilinguals With Different Age of Appropriation and Proficiency of the Second Language: A Meta-Analysis of Functional Imaging Studies , 2019, Front. Hum. Neurosci..

[42]  Liina Pylkkänen,et al.  The time-course and spatial distribution of brain activity associated with sentence processing , 2012, NeuroImage.

[43]  S. Dehaene,et al.  Cortical representation of the constituent structure of sentences , 2011, Proceedings of the National Academy of Sciences.

[44]  R. C. Oldfield.  THE ASSESSMENT AND ANALYSIS OF HANDEDNESS , 1971 .

[45]  Lars Meyer,et al.  Building by Syntax: The Neural Basis of Minimal Linguistic Structures , 2015, Cerebral cortex.

[46]  R. Malach,et al.  Syntactic structure building in the anterior temporal lobe during natural story listening , 2012, Brain and Language.

[47]  Wendy Sandler,et al.  Sign Language and Linguistic Universals: Entering the lexicon: lexicalization, backformation, and cross-modal borrowing , 2006 .

[48]  A. Lerner,et al.  Neurotrauma , 1990, Neurology.

[49]  Victoria Nyst,et al.  A descriptive analysis of Adamorobe sign language (Ghana) , 2007 .

[50]  Gregory Hickok,et al.  The Role of Broca's Area in Sentence Comprehension , 2011, Journal of Cognitive Neuroscience.

[51]  U. Bellugi,et al.  Neural Systems Mediating American Sign Language: Effects of Sensory Experience and Age of Acquisition , 1997, Brain and Language.

[52]  Matthew K. Leonard,et al.  Neural Language Processing in Adolescent First-Language Learners: Longitudinal Case Studies in American Sign Language. , 2016, Cerebral cortex.

[53]  D Bavelier,et al.  Cerebral organization for language in deaf and hearing subjects: biological constraints and effects of experience. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[54]  Naomi K. Caselli,et al.  The ASL-LEX 2.0 Project: A Database of Lexical and Phonological Properties for 2,723 Signs in American Sign Language , 2021, Journal of deaf studies and deaf education.

[55]  Eric Halgren,et al.  Neural stages of spoken, written, and signed word processing in beginning second language learners , 2013, Front. Hum. Neurosci..

[56]  Eric Halgren,et al.  Signed Words in the Congenitally Deaf Evoke Typical Late Lexicosemantic Responses with No Early Visual Responses in Left Superior Temporal Cortex , 2012, The Journal of Neuroscience.

[57]  Bencie Woll,et al.  Lexical and sentential processing in British Sign Language , 2006, Human brain mapping.

[58]  Stanislas Dehaene,et al.  Neurophysiological dynamics of phrase-structure building during sentence processing , 2017, Proceedings of the National Academy of Sciences.

[59]  C. Padden,et al.  The emergence of grammar: systematic structure in a new language. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[60]  Kuniyoshi L Sakai,et al.  Sign and speech: amodal commonality in left hemisphere dominance for comprehension of sentences. , 2005, Brain : a journal of neurology.

[61]  J. Trueswell,et al.  Cognitive control and parsing: Reexamining the role of Broca’s area in sentence comprehension , 2005, Cognitive, affective & behavioral neuroscience.

[62]  Jack A. Naglieri,et al.  Wechsler Nonverbal Scale of Ability (WNV). , 2009 .

[63]  A. Friederici.  The brain basis of language processing: from structure to function. , 2011, Physiological reviews.

[64]  Bencie Woll,et al.  Testing comprehension abilities in users of British Sign Language following CVA , 2005, Brain and Language.