Adaptive Subtitles: Preferences and Trade-Offs in Real-Time Media Adaption

Subtitles can help improve the understanding of media content. People enable subtitles based on individual characteristics (e.g., language or hearing ability), viewing environment, or media context (e.g., drama, quiz show). However, some people find that subtitles can be distracting and that they negatively impact their viewing experience. We explore the challenges and opportunities surrounding interaction with real-time personalisation of subtitled content. To understand how people currently interact with subtitles, we first conducted an online questionnaire with 102 participants. We used our findings to elicit requirements for a new approach called Adaptive Subtitles that allows the viewer to alter which speakers have subtitles displayed in real-time. We evaluated our approach with 19 participants to understand the interaction trade-offs and challenges within real-time adaptations of subtitled media. Our evaluation findings suggest that granular controls and structured onboarding allow viewers to make informed trade-offs when adapting media content, leading to improved viewing experiences.

[1]  Jason M. Silveira,et al.  The effect of subtitles on listeners’ perceptions of expressivity , 2014 .

[2]  Austin Henderson,et al.  What you see, some of what's in the future, and how we go about doing it: HI at Apple Computer , 1995, CHI 95 Conference Companion.

[3]  Krzysztof Z. Gajos,et al.  Ability-based design , 2018, Commun. ACM.

[4]  Andy Brown,et al.  Contrasting delivery modes for second screen TV content - Push or pull? , 2019, Int. J. Hum. Comput. Stud..

[5]  Frank Melchior,et al.  Creating object-based experiences in the real world , 2016 .

[6]  Krishnan Ramanathan,et al.  Personalized video: leanback online video consumption , 2011, SIGIR '11.

[7]  Wendy Fox,et al.  Integrated titles: An improved viewing experience? , 2016 .

[8]  Rhianne Jones,et al.  Online News Videos: The UX of Subtitle Position , 2015, ASSETS.

[9]  Paul Markham,et al.  Captioned Videotapes and Second‐Language Listening Word Recognition , 1999 .

[10]  Stephen Doherty,et al.  Original Language Subtitles: Their Effects on the Native and Foreign Viewer , 2017 .

[11]  David R. Flatla,et al.  It's not just the light: understanding the factors causing situational visual impairments during mobile interaction , 2018, NordiCHI.

[12]  Frank Melchior,et al.  Object-based broadcasting - curation, responsiveness and user experience , 2014 .

[13]  Danielle M. Lottridge,et al.  "Genderfluid" or "Attack Helicopter": Responsible HCI Research Practice with Non-binary Gender Variation in Online Communities , 2018, CHI.

[14]  Andy Brown,et al.  Understanding the Diverse Needs of Subtitle Users in a Rapidly Evolving Media Landscape , 2016 .

[15]  Wijnand A. IJsselsteijn,et al.  The impact of video characteristics and subtitles on visual comfort of 3D TV , 2013, Displays.

[16]  Rhianne Jones,et al.  Responsive design for personalised subtitles , 2015, W4A.

[17]  Andrea Forte,et al.  Reliability and Inter-rater Reliability in Qualitative Research , 2019, Proc. ACM Hum. Comput. Interact..

[18]  Sarah J. Tracy.  Qualitative Research Methods: Collecting Evidence, Crafting Analysis, Communicating Impact , 2012 .

[19]  Andy Brown,et al.  Subtitles in 360-degree Video , 2017, TVX.

[20]  Vicki L. Hanson,et al.  A web accessibility service: update and findings , 2004, Assets '04.

[21]  David R. Flatla,et al.  MirrorMirror: A Mobile Application to Improve Speechreading Acquisition , 2018, CHI.

[22]  Rachel Menzies,et al.  Emoji Accessibility for Visually Impaired People , 2020, CHI.

[23]  Pilar Orero,et al.  Usability tests for personalised subtitles , 2018, Translation Spaces.

[24]  Olivier Penacchio,et al.  Visual attention to subtitles when viewing a cartoon by deaf and hearing children: an eye-tracking pilot study , 2014 .

[25]  Peter Gregor,et al.  Accessible accessibility , 2000, CUU '00.

[26]  Rhianne Jones,et al.  Developing Accessible Services: Understanding Current Knowledge and Areas for Future Support , 2019, CHI.

[27]  Helen E. Katz.  The Media Handbook : A Complete Guide to Advertising Media Selection, Planning, Research, and Buying , 2019 .

[28]  Andy Brown,et al.  Dynamic Subtitles: The User Experience , 2015, TVX.

[29]  David Sloan,et al.  The potential of adaptive interfaces as an accessibility aid for older web users , 2010, W4A.

[30]  Kyle Montague,et al.  Designing for individuals: usable touch-screen interaction through shared user models , 2012, ASSETS '12.

[31]  Mike Armstrong.  Automatic recovery and verification of subtitles for large collections of video clips , 2016 .

[32]  Robert Burch,et al.  Time Spent Viewing Captions on Television Programs , 2000, American annals of the deaf.

[33]  Denis Burnham,et al.  Parameters in television captioning for deaf and hard-of-hearing adults: effects of caption rate versus text reduction on comprehension. , 2008, Journal of deaf studies and deaf education.

[34]  Rhianne Jones,et al.  The Development of a Framework for Understanding the UX of Subtitles , 2015, ASSETS.

[35]  Chris J. Hughes,et al.  Disruptive Approaches for Subtitling in Immersive Environments , 2019, TVX.

[36]  Patrick Le Callet,et al.  Visual attention as a dimension of QoE: Subtitles in UHD videos , 2016, 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX).

[37]  J. McQueen,et al.  Foreign Subtitles Help but Native-Language Subtitles Harm Foreign Speech Perception , 2009, PloS one.

[38]  Agnieszka Szarkowska,et al.  Subtitles on the moving image: an overview of eye tracking studies , 2015 .

[39]  D. Ewoldsen,et al.  The Impact of Subtitles on Comprehension of Narrative Film , 2013 .

[40]  David R. Flatla,et al.  A Framework for Speechreading Acquisition Tools , 2017, CHI.

[41]  Sergio Firmenich,et al.  Personalized Web Accessibility using Client-Side Refactoring , 2013, IEEE Internet Computing.