The impact of text segmentation on subtitle reading

Understanding the way people watch subtitled films has become a central concern for subtitling researchers in recent years. Both subtitling scholars and professionals generally believe that in order to reduce cognitive load and enhance readability, line breaks in two-line subtitles should follow syntactic units. However, previous research has been inconclusive as to whether syntactic-based segmentation facilitates comprehension and reduces cognitive load. In this study, we assessed the impact of text segmentation on subtitle processing among different groups of viewers: hearing people with different mother tongues (English, Polish, and Spanish) and deaf, hard of hearing, and hearing people with English as a first language. We measured three indicators of cognitive load (difficulty, effort, and frustration) as well as comprehension and eye tracking variables.  Participants watched two video excerpts with syntactically and non-syntactically segmented subtitles. The aim was to determine whether syntactic-based text segmentation as well as the viewers’ linguistic background influence subtitle processing. Our findings show that non-syntactically segmented subtitles induced higher cognitive load, but they did not adversely affect comprehension. The results are discussed in the context of cognitive load, audiovisual translation, and deafness.

[1]  Krzysztof Krejtz,et al.  Cognitive load in intralingual and interlingual respeaking – a preliminary study , 2016 .

[2]  Elisa Perego,et al.  Subtitles and line-breaks: towards improved readability , 2008 .

[3]  Amy M. Lieberman,et al.  Achievement in Relation to Phonological Coding and Awareness in Deaf Readers : A Meta-analysis , 2011 .

[4]  Jan-Louis Kruger,et al.  Measuring the impact of subtitles on cognitive load: eye tracking and dynamic audiovisual texts , 2013, ETSA '13.

[5]  Wendy Fox,et al.  Multimodal measurement of cognitive load during subtitle processing: same-language subtitles for foreign-language viewers , 2018 .

[6]  Stephen Doherty,et al.  The development of eye tracking in empirical research on subtitling and captioning , 2018 .

[7]  Carol Musselman,et al.  How Do Children Who Can't Hear Learn to Read an Alphabetic Script? A Review of the Literature on Reading and Deafness. , 2000, Journal of deaf studies and deaf education.

[8]  Stacey A. Keenan,et al.  Effects of Chunking and Line Length on Reading Efficiency. , 1984 .

[9]  I. Krejtz,et al.  Reading Function and Content Words in Subtitled Videos. , 2016, Journal of deaf studies and deaf education.

[10]  Carl Jensema.  Viewer Reaction to Different Television Captioning Speeds , 1998, American annals of the deaf.

[11]  Frank Thorn,et al.  Television Captions for Hearing-Impaired People: A Study of Key Factors that Affect Reading Performance , 1996, Hum. Factors.

[12]  Elisa Perego,et al.  The Cognitive Effectiveness of Subtitle Processing , 2010 .

[13]  Lyn Frazier,et al.  ON COMPREHENDING SENTENCES: SYNTACTIC PARSING STRATEGIES. , 1979 .

[14]  G. Belenky,et al.  Investigating the temporal dynamics and underlying mechanisms of cognitive fatigue. , 2011 .

[15]  R. Tunney,et al.  Processing of native and foreign language subtitles in films: An eye tracking study , 2012, Applied Psycholinguistics.

[16]  P. Ackerman,et al.  Test length and cognitive fatigue: an empirical examination of effects on performance and test-taker reactions. , 2009, Journal of experimental psychology. Applied.

[17]  F. Paas,et al.  Cognitive Load Measurement as a Means to Advance Cognitive Load Theory , 2003 .

[18]  P. Chandler,et al.  Cognitive Load Theory and the Format of Instruction , 1991 .

[19]  Géry d'Ydewalle,et al.  Lengthening the Presentation Time of Subtitles on Television: Effects on Children’s Reading Time and Recognition , 1999 .

[20]  Alan Kennedy,et al.  Book Review: Eye Tracking: A Comprehensive Guide to Methods and Measures , 2016, Quarterly journal of experimental psychology.

[21]  F. Paas,et al.  Memory load and the cognitive pupillary response in aging. , 2004, Psychophysiology.

[22]  F. Paas,et al.  Instructional Efficiency: Revisiting the Original Construct in Educational Research , 2008 .

[23]  Jan-Louis Kruger,et al.  Subtitles and Eye Tracking: Reading and Performance , 2014 .

[24]  C. Reid,et al.  Parsing Complements: Comments on the Generality of the Principle of Minimal Attachment , 1989 .

[25]  Géry d'Ydewalle,et al.  READING A MESSAGE WHEN THE SAME MESSAGE IS AVAILABLE AUDITORILY IN ANOTHER LANGUAGE: THE CASE OF SUBTITLING , 1987 .

[26]  Paul Macaruso,et al.  Syntactically cued text facilitates oral reading fluency in developing readers , 2006, Applied Psycholinguistics.

[27]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[28]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[29]  Pilar Orero,et al.  E ff ects of Text Chunking on Subtitling: A Quantitative and Qualitative Examination , 2011 .

[30]  Agnieszka Szarkowska,et al.  Subtitles on the moving image: an overview of eye tracking studies , 2015 .

[31]  Pablo Romero-Fresco,et al.  The Reception of Subtitles for the Deaf and Hard of Hearing in Europe , 2016 .

[32]  Agnieszka Szarkowska,et al.  The Effects of Shot Changes on Eye Movements in Subtitling , 2013 .

[33]  C. Jensema,et al.  Eye Movement Patterns of Captioned Television Viewers , 2000, American annals of the deaf.

[34]  Jorge Díaz-Cintas,et al.  Audiovisual Translation, Subtitling , 2014 .

[35]  Margarita Kaushanskaya,et al.  The Language Experience and Proficiency Questionnaire (LEAP-Q): assessing language profiles in bilinguals and multilinguals. , 2007, Journal of speech, language, and hearing research : JSLHR.

[36]  Zuzanna Klyszejko,et al.  Verbatim, Standard, or Edited?: Reading Patterns of Different Captioning Styles Among Deaf, Hard of Hearing, and Hearing Viewers , 2011, American annals of the deaf.

[37]  Stephen Doherty,et al.  Measuring Cognitive Load in the Presence of Educational Video: Towards a Multimodal Methodology. , 2016 .

[38]  Donald Mitchell,et al.  Lexical guidance in human parsing: Locus and processing characteristics. , 1987 .

[39]  F. Paas,et al.  Cognitive Architecture and Instructional Design , 1998 .

[40]  G. Wylie,et al.  Subjective Cognitive Fatigue in Multiple Sclerosis Depends on Task Length , 2014, Front. Neurol..

[41]  Pavlo D. Antonenko,et al.  Using Electroencephalography to Measure Cognitive Load , 2010 .

[42]  Brittany R. L. Duff,et al.  All Loads Are Not Equal: Distinct Influences of Perceptual Load and Cognitive Load on Peripheral Ad Processing , 2016 .

[43]  Jan-Louis Kruger,et al.  Attention distribution and cognitive load in a subtitled academic lecture: L1 vs. L2 , 2014 .

[44]  Don C. Mitchell,et al.  Verb guidance and other lexical effects in parsing , 1989 .

[45]  I. Dey.  What is qualitative data , 1993 .

[46]  Marta Miquel Iriarte.  The reception of subtitling for the deaf and hard of hearing: viewers’ hearing and communication profile & Subtitling speed of exposure , 2017 .

[47]  Ian Calvert,et al.  Reading in multimodal environments: assessing legibility and accessibility of typography for television , 2011, ASSETS '11.

[48]  Agnieszka Szarkowska,et al.  The effects of text editing and subtitle presentation rate on the comprehension and reading patterns of interlingual and intralingual subtitles among deaf, hard of hearing and hearing viewers , 2016 .

[49]  A. Baddeley.  Working Memory, Thought, and Action , 2007 .

[50]  J. Yoon,et al.  The Effects of Captions on Deaf Students' Content Comprehension, Cognitive Load, and Motivation in Online Learning , 2011, American annals of the deaf.

[51]  T. Gog,et al.  Measuring cognitive load with subjective rating scales during problem solving: differences between immediate and delayed ratings , 2015 .

[52]  Alexander Pollatsek,et al.  Psychology of reading, 2nd ed. , 2012 .

[53]  K. Verfaillie,et al.  Watching Subtitled Television , 1991 .

[54]  A. Remael,et al.  Is subtitling equally effective everywhere? A first cross-national study on the reception of interlingually subtitled messages , 2016 .

[55]  Morton Ann Gernsbacher,et al.  Video Captions Benefit Everyone , 2015, Policy insights from the behavioral and brain sciences.

[56]  Ralph Radach,et al.  Eye Movements in Reading , 1999 .

[57]  Gregory M. Shreve,et al.  Measuring translation difficulty: An empirical study , 2014 .

[58]  John Sweller,et al.  Cognitive Load Theory , 2020, Encyclopedia of Education and Information Technologies.

[59]  Géry d'Ydewalle,et al.  Eye Movements of Children and Adults While Reading Television Subtitles , 2007 .