Factors Influencing the Use of Captions by Foreign Language Learners: An Eye‐Tracking Study

This study investigates caption-reading behavior by foreign language (L2) learners and, through eye-tracking methodology, explores the extent to which the relationship between the native and target language affects that behavior. Second-year (4th semester) English-speaking learners of Arabic, Chinese, Russian, and Spanish watched 2 videos differing in content familiarity, each dubbed and captioned in the target language. Results indicated that time spent on captions differed significantly by language: Arabic learners spent more time on captions than learners of Spanish and Russian. A significant interaction between language and content familiarity occurred: Chinese learners spent less time on captions in the unfamiliar content video than the familiar, while others spent comparable times on each. Based on dual-processing and cognitive load theories, we posit that the Chinese learners experienced a split-attention effect when verbal processing was difficult and that, overall, captioning benefits during the 4th semester of language learning are constrained by L2 differences, including differences in script, vocabulary knowledge, concomitant L2 proficiency, and instructional methods. Results are triangulated with qualitative findings from interviews. [ABSTRACT FROM AUTHOR]

[1]  Michael Harrington.  Cognition and Second Language Instruction: Sentence processing , 2001 .

[2]  M A Just,et al.  A theory of reading: from eye fixations to comprehension. , 1980, Psychological review.

[3]  Keith Rayner,et al.  The influence of text legibility on eye movements during reading , 2010 .

[4]  P. Chandler,et al.  Cognitive Load Theory and the Format of Instruction , 1991 .

[5]  Martine Danan,et al.  Reversed subtitling and dual coding theory: New directions for foreign language instruction , 1992 .

[6]  Alexander V. Kravchenko,et al.  The experiential basis of speech and writing as different cognitive domains , 2009 .

[7]  Gregory Taylor,et al.  Perceived Processing Strategies of Students Watching Captioned Video , 2005 .

[8]  K. Rayner,et al.  Contextual effects on word perception and eye movements during reading , 1981 .

[9]  Nuria Sagarra,et al.  FROM SEEING ADVERBS TO SEEING VERBAL MORPHOLOGY , 2013, Studies in Second Language Acquisition.

[10]  Geoffrey Underwood,et al.  Inspecting Pictures for Information to Verify a Sentence: Eye Movements in General Encoding and in Focused Search , 2004, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[11]  Nuria Sagarra,et al.  Learned Attention Effects in L2 Temporal Reference: The First Hour and the Next Eight Semesters , 2010 .

[12]  Iva Baltova,et al.  Multisensory language teaching in a multidimensional curriculum : The use of authentic bimodal video in core French , 1999 .

[13]  Bryan Smith,et al.  Employing Eye-Tracking Technology in Researching the Effectiveness of Recasts in CMC , 2010 .

[14]  Nick C. Ellis,et al.  LEARNED ATTENTION IN ADULT LANGUAGE ACQUISITION , 2011, Studies in Second Language Acquisition.

[15]  Patricia Dunkel,et al.  The Effect of Speech Modification, Prior Knowledge, and Listening Proficiency on EFL Lecture Learning , 1992 .

[16]  A. Weber,et al.  Lexical competition in non-native spoken-word recognition , 2004 .

[17]  Tetyana Sydorenko,et al.  Modality of Input and Vocabulary Acquisition , 2010 .

[18]  Cheryl Frenck-Mestre,et al.  Eye-movement recording as a tool for studying syntactic processing in a second language: a review of methodologies and experimental findings , 2005 .

[19]  David E. Eskey,et al.  The Effects of Closed-Captioned Television on the Listening Comprehension of Intermediate English as a Second Language (ESL) Students , 1999 .

[20]  Keith Rayner,et al.  Children's and Adults’ Processing of Anomaly and Implausibility during Reading: Evidence from Eye Movements , 2008, Quarterly journal of experimental psychology.

[21]  Erik D. Reichle,et al.  Using E-Z reader to model the effects of higher level language processing on eye movements during reading , 2009, Psychonomic bulletin & review.

[22]  Thomas J. Garza,et al.  Evaluating the Use of Captioned Video Materials in Advanced Foreign Language Learning , 1991 .

[23]  Marianne Gullberg,et al.  ONLINE PRONOUN RESOLUTION IN L2 DISCOURSE: L1 Influence and General Learner Effects , 2008, Studies in Second Language Acquisition.

[24]  C. Goh.  A cognitive perspective on language learners' listening comprehension problems , 2000 .

[25]  Rosemary Erlam,et al.  The Effect and the Influence of the Use of Video and Captions on Second Language Learning , 2008 .

[26]  Bryan Smith,et al.  Eye tracking as a measure of noticing: A study of explicit recasts in SCMC , 2012 .

[27]  Martine Danan,et al.  Captioning and Subtitling: Undervalued Language Learning Strategies , 2004 .

[28]  Myongsu Park.  The effects of partial captions on Korean EFL learners' listening comprehension , 2004 .

[29]  Takashi Otake,et al.  Asymmetric mapping from phonetic to lexical representations in second-language listening , 2006, J. Phonetics.

[30]  Robert Vanderplank,et al.  Déjà vu? A decade of research on language laboratories, television and video in language learning , 2009, Language Teaching.

[31]  Bert H. Hodges.  Good Prospects: Ecological and Social Perspectives on Conforming, Creating, and Caring in Conversation. , 2007 .

[32]  Paul Markham,et al.  Captioned Videotapes and Second‐Language Listening Word Recognition , 1999 .

[33]  Patti Spinner,et al.  ECOLOGICAL VALIDITY IN EYE-TRACKING , 2013, Studies in Second Language Acquisition.

[34]  Paul Gruba,et al.  Understanding Digitized Second Language Videotext , 2004 .

[35]  Paola E. Dussias.  Uses of Eye-Tracking Data in Second Language Sentence Processing Research , 2010, Annual Review of Applied Linguistics.

[36]  Irshad Hussain,et al.  Role of Mobile Technology in Promoting Campus-Wide Learning Environment , 2009 .

[37]  Joël Pynte,et al.  Reading as a Perceptual Process , 2000 .

[38]  G. L. Riley,et al.  A comparison of recall and summary protocols as measures of second language reading comprehension , 1996 .

[39]  Jae Youn Hwang,et al.  Novel algorithm for improving accuracy of ultrasound measurement of residual urine volume according to bladder shape. , 2004, Urology.

[40]  M. Just,et al.  Eye fixations and cognitive processes , 1976, Cognitive Psychology.

[41]  Peter Shea,et al.  Leveling the Playing Field: A Study of Captioned Interactive Video for Second Language Learning , 2000 .

[42]  Géry d’Ydewalle,et al.  Attention Allocation with Overlapping Sound, Image, and Text , 1992 .

[43]  Paula M. Winke,et al.  INTRODUCTION TO THE SPECIAL ISSUE , 2013, Studies in Second Language Acquisition.

[44]  Erik D. Reichle,et al.  The E-Z Reader model of eye-movement control in reading: Comparisons to other models , 2003, Behavioral and Brain Sciences.

[45]  Lars Stenius Stæhr.  VOCABULARY KNOWLEDGE AND ADVANCED LISTENING COMPREHENSION IN ENGLISH AS A FOREIGN LANGUAGE , 2009, Studies in Second Language Acquisition.

[46]  Josh Ard,et al.  Verification of Language Transfer , 1992 .

[47]  K. Rayner,et al.  Toward a model of eye movement control in reading. , 1998 .

[48]  K. Verfaillie,et al.  Watching Subtitled Television , 1991 .

[49]  Sue E. K. Otto,et al.  Applying Instructional Technologies , 1990 .

[50]  J. Grainger,et al.  Recognition of Cognates and Interlingual Homographs: The Neglected Role of Phonology , 1999 .

[51]  Alexander Pollatsek,et al.  Extending the E-Z Reader Model of Eye Movement Control to Chinese Readers , 2007, Cogn. Sci..

[52]  Don C. Mitchell,et al.  Accounting for regressive eye-movements in models of sentence processing: A reappraisal of the Selective Reanalysis hypothesis , 2008 .

[53]  R. Mayer,et al.  A Split-Attention Effect in Multimedia Learning: Evidence for Dual Processing Systems in Working Memory , 1998 .

[54]  Debra Hoven,et al.  A model for listening and viewing comprehension in multimedia environments , 1999 .

[55]  A. Kravchenko,et al.  Essential properties of language, or, why language is not a code , 2007 .

[56]  Paul Markham,et al.  The Influence of Culture-Specific Background Knowledge and Captions on Second Language Comprehension , 2001 .

[57]  Paul Gruba,et al.  PLAYING THE VIDEOTEXT: A MEDIA LITERACY PERSPECTIVE ON VIDEO-MEDIATED L2 LISTENING , 2006 .

[58]  Jurgen Froehlich.  German Videos with German Subtitles: A New Approach to Listening Comprehension Development. , 1988 .

[59]  Tessa Warren,et al.  Investigating effects of selectional restriction violations and plausibility violation severity on eye-movements in reading , 2007, Psychonomic bulletin & review.

[60]  R. Leow.  A STUDY OF THE ROLE OF AWARENESS IN FOREIGN LANGUAGE BEHAVIOR , 2000, Studies in Second Language Acquisition.

[61]  J. Chung,et al.  The Effects of Using Video Texts Supported with Advance Organizers and Captions on Chinese College Students' Listening Comprehension: An Empirical Study , 1999 .

[62]  T. J. McCarthy,et al.  The Effects of Native Language vs. Target Language Captions on Foreign Language Students' DVD Video Comprehension , 2001 .

[63]  K. Rayner,et al.  Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity , 1986, Memory & cognition.

[64]  R. Tunney,et al.  Processing of native and foreign language subtitles in films: An eye tracking study , 2012, Applied Psycholinguistics.

[65]  K. Rayner.  Eye Movements in Reading: Models and Data. , 2009 .

[66]  Lars Placke,et al.  Eye movements during the reading of compound words and the influence of lexeme meaning , 2008, Memory & cognition.

[67]  Paul Markham,et al.  Captioned Television Videotapes: Effects of Visual Support on Second Language Comprehension , 1993 .

[68]  Ellen Bialystok,et al.  Bilingualism, Biliteracy, and Learning to Read: Interactions Among Languages and Writing Systems , 2005 .

[69]  Erik D. Reichle,et al.  Tests of the E-Z Reader model: Exploring the interface between cognition and eye-movement control , 2006, Cognitive Psychology.

[70]  Barbara J. Juhasz,et al.  The effect of plausibility on eye movements in reading. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[71]  Suzanne Graham,et al.  Listening comprehension: The learners’ perspective , 2006 .

[72]  Paul Markham,et al.  The Influence of English Language and Spanish Language Captions on Foreign Language Listening/Reading Comprehension , 2003 .

[73]  Andrew J. Stewart,et al.  Integrating text and pictorial information: eye movements when looking at print advertisements. , 2001, Journal of experimental psychology. Applied.

[74]  Michael J. Spivey,et al.  Shared and separate systems in bilingual language processing: Converging evidence from eyetracking and brain imaging , 2003, Brain and Language.

[75]  Paola E. Dussias.  Parsing a first language like a second: The erosion of L1 parsing strategies in Spanish-English Bilinguals , 2004 .

[76]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[77]  R. Mayer,et al.  Verbal redundancy in multimedia learning: When reading helps listening , 2002 .

[78]  Aline Godfroid,et al.  ATTENTION TO IRREGULAR VERBS BY BEGINNING LEARNERS OF GERMAN , 2013, Studies in Second Language Acquisition.

[79]  Robert Vanderplank,et al.  The value of teletext sub-titles in language learning , 1988 .

[80]  Géry d'Ydewalle,et al.  Eye Movements of Children and Adults While Reading Television Subtitles , 2007 .

[81]  Joseph R. Weyers.  The Effect of Authentic Video on Communicative Competence , 1999 .

[82]  D. R. Long.  What You Don't Know Can't Help You , 1990, Studies in Second Language Acquisition.

[83]  Paula M. Winke,et al.  THE EFFECTS OF INPUT ENHANCEMENT ON GRAMMAR LEARNING AND COMPREHENSION , 2013, Studies in Second Language Acquisition.

[84]  Patricia S. Koskinen,et al.  Captioned Television as Comprehensible Input: Effects of Incidental Word Learning from Context for Language Minority Students. , 1992 .

[85]  F. Boers,et al.  AN EYE FOR WORDS , 2013, Studies in Second Language Acquisition.

[86]  John N. Williams,et al.  The effect of bimodal input on implicit and explicit memory: An investigation into the benefits of within-language subtitling , 2002 .

[87]  R. Hartsuiker,et al.  The influence of semantic constraints on bilingual word recognition during sentence reading , 2011 .

[88]  Helen Gant Guillory,et al.  The Effects of Keyword Captions to Authentic French Video on Learner Comprehension , 2013 .

[89]  Géry d’Ydewalle,et al.  Could Enjoying a Movie Lead to Language Acquisition , 1997 .

[90]  J. Plass,et al.  Supporting Listening Comprehension and Vocabulary Acquisition in French with Multimedia Annotations , 2002 .