Bimodal bilinguals co-activate both languages during spoken comprehension

Bilinguals have been shown to activate their two languages in parallel, and this process can often be attributed to overlap in input between the two languages. The present study examines whether two languages that do not overlap in input structure, and that have distinct phonological systems, such as American Sign Language (ASL) and English, are also activated in parallel. Hearing ASL-English bimodal bilinguals' and English monolinguals' eye-movements were recorded during a visual world paradigm, in which participants were instructed, in English, to select objects from a display. In critical trials, the target item appeared with a competing item that overlapped with the target in ASL phonology. Bimodal bilinguals looked more at competing item than at phonologically unrelated items and looked more at competing items relative to monolinguals, indicating activation of the sign-language during spoken English comprehension. The findings suggest that language co-activation is not modality specific, and provide insight into the mechanisms that may underlie cross-modal language co-activation in bimodal bilinguals, including the role that top-down and lateral connections between levels of processing may play in language comprehension.

[1]  J. Ziegler,et al.  Neighborhood effects in auditory word recognition: Phonological competition and orthographic facilitation. , 2003 .

[2]  Michael J. Spivey,et al.  Cross Talk Between Native and Second Languages: Partial Activation of an Irrelevant Lexicon , 1999 .

[3]  P. Viviani Eye movements in visual search: cognitive, perceptual and motor control aspects. , 1990, Reviews of oculomotor research.

[4]  Viorica Marian,et al.  Audio-visual integration during bilingual language processing , 2009 .

[5]  Lori L. Holt,et al.  Are there interactive processes in speech perception? , 2006, Trends in Cognitive Sciences.

[6]  Angeliki Salamoura,et al.  Processing verb argument structure across languages: Evidence for shared representations in the bilingual lexicon , 2007, Applied Psycholinguistics.

[7]  Paola E. Dussias,et al.  The effect of exposure on syntactic parsing in Spanish–English bilinguals , 2007, Bilingualism: Language and Cognition.

[8]  A. Weber,et al.  Lexical competition in non-native spoken-word recognition , 2004 .

[9]  S. Giaquinto,et al.  Stability of word comprehension with age An electrophysiological study , 2007, Mechanisms of Ageing and Development.

[10]  D. Norris,et al.  No lexical–prelexical feedback during speech perception or: Is it time to stop playing those Christmas tapes? , 2009 .

[11]  Takashi Otake,et al.  Asymmetric mapping from phonetic to lexical representations in second-language listening , 2006, J. Phonetics.

[12]  Marc Brysbaert,et al.  Visual word recognition in bilinguals: phonological priming from the second to the first language. , 2002, Journal of experimental psychology. Human perception and performance.

[13]  Andrea Weber,et al.  Finding Referents in Time: Eye-Tracking Evidence for the Role of Contrastive Accents , 2006, Language and speech.

[14]  J. Hoffmann,et al.  The European Society for Cognitive Psychology , 1999 .

[15]  Julie C. Sedivy,et al.  Eye movements to pictures reveal transient semantic activation during spoken word recognition. , 2006, Journal of experimental psychology. Learning, memory, and cognition.

[16]  G. Altmann,et al.  Word meaning and the control of eye fixation: semantic competitor effects and the visual world paradigm , 2005, Cognition.

[17]  Ana I. Schwartz,et al.  Bilingual lexical activation in sentence context , 2006 .

[18]  J. Kroll,et al.  FIRST LANGUAGE ACTIVATION DURING SECOND LANGUAGE LEXICAL PROCESSING: An Investigation of Lexical Form, Meaning, and Grammatical Class , 2006, Studies in Second Language Acquisition.

[19]  Marc Brys,et al.  Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English , 2009 .

[20]  T. Dijkstra,et al.  The architecture of the bilingual word recognition system: From identification to decision , 2002, Bilingualism: Language and Cognition.

[21]  James S. Magnuson,et al.  Immediate effects of form-class constraints on spoken word recognition , 2008, Cognition.

[22]  G. Thierry,et al.  Brain potentials reveal unconscious translation during foreign-language comprehension , 2007, Proceedings of the National Academy of Sciences.

[23]  J. Kroll,et al.  Tutorials in bilingualism : psycholinguistic perspectives , 1999 .

[24]  Judith F. Kroll,et al.  When deaf signers read English: Do written words activate their sign translations? , 2011, Cognition.

[25]  Alessandro Angrilli,et al.  Developmental aspects of automatic word processing: Language lateralization of early ERP components in children, young adults and middle-aged subjects , 2009, Biological Psychology.

[26]  Paul D. Allopenna,et al.  Tracking the Time Course of Spoken Word Recognition Using Eye Movements: Evidence for Continuous Mapping Models , 1998 .

[27]  Eva Belke,et al.  Early activation of object names in visual search , 2007, Psychonomic bulletin & review.

[28]  Margarita Kaushanskaya,et al.  Bilingual Language Processing and Interference in Bilinguals: Evidence From Eye Tracking and Picture Naming , 2007 .

[29]  Karen Emmorey,et al.  Bimodal bilingualism. , 2008, Bilingualism.

[30]  M. Pickering,et al.  The representation of lexical and syntactic information in bilinguals: Evidence from syntactic priming , 2007 .

[31]  Julie C. Sedivy,et al.  Subject Terms: Linguistics Language Eyes & eyesight Cognition & reasoning , 1995 .

[32]  U. Bellugi,et al.  Neural Systems Mediating American Sign Language: Effects of Sensory Experience and Age of Acquisition , 1997, Brain and Language.

[33]  J. Navarra,et al.  Hearing lips in a second language: visual articulatory information enables the perception of second language sounds , 2007, Psychological research.

[34]  M. Pickering,et al.  Is Syntax Separate or Shared Between Languages? , 2004, Psychological science.

[35]  Margarita Kaushanskaya,et al.  The Language Experience and Proficiency Questionnaire (LEAP-Q): assessing language profiles in bilinguals and multilinguals. , 2007, Journal of speech, language, and hearing research : JSLHR.

[36]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[37]  P. Luce,et al.  Falling on Sensitive Ears , 2004, Psychological science.

[38]  K. Forster,et al.  The role of polysemy in masked semantic and translation priming , 2004 .

[39]  Antoine J. Shahin,et al.  Multisensory integration enhances phonemic restoration. , 2009, The Journal of the Acoustical Society of America.

[40]  Katie Wagner,et al.  Carpet or Cárcel: The effect of age of acquisition and language mode on bilingual lexical access , 2010 .

[41]  Listening comprehension : approach , design , and procedure , 2022 .

[42]  Q. Summerfield Some preliminaries to a comprehensive account of audio-visual speech perception. , 1987 .

[43]  Diane Brentari,et al.  A Prosodic Model of Sign Language Phonology , 1999 .

[44]  Michael J. Spivey,et al.  Bilingual and monolingual processing of competing lexical items , 2003, Applied Psycholinguistics.

[45]  Marc Brysbaert,et al.  Semantic and translation priming from a first language to a second and back: Making sense of the findings , 2009, Memory & cognition.

[46]  Ulrich H. Frauenfelder,et al.  Bidirectional grapheme-phoneme activation in a bimodal detection task. , 1993, Journal of experimental psychology. Human perception and performance.

[47]  Falk Huettig,et al.  The tug of war between phonological, semantic and shape information in language-mediated visual search , 2007 .

[48]  V. Marian,et al.  Constraints on parallel activation in bilingual spoken language processing: Examining proficiency and lexical status using eye-tracking , 2007 .

[49]  Margarita Kaushanskaya,et al.  Nontarget language recognition and interference in bilinguals: Evidence from eye-tracking and picture naming (Language Learning 57, 1, (119-163)) , 2008 .

[50]  James Bartolotti,et al.  Language Learning and Control in Monolinguals and Bilinguals , 2012, Cogn. Sci..

[51]  Michael J. Spivey,et al.  Competing activation in bilingual language processing: Within- and between-language competition , 2003, Bilingualism: Language and Cognition.

[52]  J. Grainger,et al.  Orthographic neighborhood effects in bilingual word recognition , 1998 .

[53]  A. Goldstein,et al.  The effect of aging on event-related potentials and behavioral responses: Comparison of tonal, phonologic and semantic targets , 2006, Clinical Neurophysiology.

[54]  W. Marslen-Wilson Functional parallelism in spoken word-recognition , 1987, Cognition.

[55]  V. Marian,et al.  Sensitivity to Phonological Similarity Within and Across Languages , 2008, Journal of psycholinguistic research.

[56]  J. Grainger,et al.  MASKED PRIMING BY TRANSLATION EQUIVALENTS IN PROFICIENT BILINGUALS , 1998 .

[57]  Jonathan M. Campbell,et al.  Peabody Picture Vocabulary Test , 2010 .

[58]  W. Heij,et al.  Nonverbal Context Effects in Forward and Backward Word Translation: Evidence for Concept Mediation , 1996 .

[59]  Karen Emmorey,et al.  Co-speech gesture in bimodal bilinguals , 2009, Language and cognitive processes.

[60]  J. Kroll,et al.  Lexical and conceptual memory in the bilingual: mapping form to meaning in two languages , 2020, The Bilingualism Reader.