Social, Environmental, and Technical: Factors at Play in the Current Use and Future Design of Small-Group Captioning

Real-time captioning is a critical accessibility tool for many d/Deaf and hard of hearing (DHH) people. While the vast majority of captioning work has focused on formal settings and technical innovations, in contrast, we investigate captioning for informal, interactive small-group conversations, which have a high degree of spontaneity and foster dynamic social interactions. This paper reports on semi-structured interviews and design probe activities we conducted with 15 DHH participants to understand their use of existing real-time captioning services and future design preferences for both in-person and remote small-group communication. We found that our participants' experiences of captioned small-group conversations are shaped by social, environmental, and technical considerations (e.g., interlocutors' pre-established relationships, the type of captioning displays available, and how far captions lag behind speech). When considering future captioning tools, participants were interested in greater feedback on non-speech elements of conversation (e.g., speaker identity, speech rate, volume) both for their personal use and to guide hearing interlocutors toward more accessible communication. We contribute a qualitative account of DHH people's real-time captioning experiences during small-group conversation and future design considerations to better support the groups being captioned, both in person and online.?

[1]  Benjamin M. Gorman,et al.  Adaptive Subtitles: Preferences and Trade-Offs in Real-Time Media Adaption , 2021, CHI.

[2]  Matt Huenerfauth,et al.  Deaf and hard-of-hearing users' preferences for hearing speakers' behavior during technology-mediated in-person and remote conversations , 2021, W4A.

[3]  Jeffrey P. Bigham,et al.  Say It All: Feedback for Improving Non-Visual Presentation Accessibility , 2021, CHI.

[4]  Jon E. Froehlich,et al.  What Do We Mean by “Accessibility Research”?: A Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019 , 2021, CHI.

[5]  Raja S. Kushalnagar,et al.  Teleconference Accessibility and Guidelines for Deaf and Hard of Hearing Users , 2020, International ACM SIGACCESS Conference on Computers and Accessibility.

[6]  Lefteris Moussiades,et al.  Α Benchmarking of IBM, Google and Wit Automatic Speech Recognition Systems , 2020, AIAI.

[7]  Matt Huenerfauth,et al.  Deaf Individuals' Views on Speaking Behaviors of Hearing Peers when Using an Automatic Captioning App , 2020, CHI Extended Abstracts.

[8]  Jon Froehlich,et al.  Evaluating Smartwatch-based Sound Feedback for Deaf and Hard-of-hearing Users Across Contexts , 2020, CHI.

[9]  Brian P. Trager,et al.  Exploration of Automatic Speech Recognition for Deaf and Hard of Hearing Students in Higher Education Classes , 2019, ASSETS.

[10]  Virginia Braun,et al.  Reflecting on reflexive thematic analysis , 2019, Qualitative Research in Sport, Exercise and Health.

[11]  Larwan Berke,et al.  Preferred Appearance of Captions Generated by Automatic Speech Recognition for Deaf and Hard-of-Hearing Viewers , 2019, CHI Extended Abstracts.

[12]  Matt Huenerfauth,et al.  Behavioral Changes in Speakers who are Automatically Captioned in Meetings with Deaf or Hard-of-Hearing Peers , 2018, ASSETS.

[13]  Erin Brady,et al.  Interdependence as a Frame for Assistive Technology Research and Design , 2018, ASSETS.

[14]  Shari Trewin,et al.  Leveraging Pauses to Improve Video Captions , 2018, ASSETS.

[15]  Jon Froehlich,et al.  Towards Accessible Conversations in a Mobile Context for People who are Deaf and Hard of Hearing , 2018, ASSETS.

[16]  Gábor Gosztolya,et al.  User-centric Evaluation of Automatic Punctuation in ASR Closed Captioning , 2018, INTERSPEECH.

[17]  Mike Y. Chen,et al.  SpeechBubbles: Enhancing Captioning Experiences for Deaf and Hard-of-Hearing People in Group Conversations , 2018, CHI.

[18]  Larwan Berke,et al.  Methods for Evaluation of Imperfect Captioning Tools by Deaf or Hard-of-Hearing Users at Different Reading Literacy Levels , 2018, CHI.

[19]  James A. Landay,et al.  Comparing Speech and Keyboard Text Entry for Short Messages in Two Languages on Touchscreen Phones , 2016, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[20]  Lisa B. Elliot,et al.  User Experiences When Testing a Messaging App for Communication Between Individuals who are Hearing and Deaf or Hard of Hearing , 2017, ASSETS.

[21]  Lisa B. Elliot,et al.  Personal Perspectives on Using Automatic Speech Recognition to Facilitate Communication between Deaf Students and Hearing Customers , 2017, ASSETS.

[22]  Larwan Berke,et al.  Deaf and Hard-of-Hearing Perspectives on Imperfect Automatic Speech Recognition for Captioning One-on-One Meetings , 2017, ASSETS.

[23]  Raja S. Kushalnagar,et al.  Feasibility of Using Automatic Speech Recognition with Voices of Deaf and Hard-of-Hearing Individuals , 2017, ASSETS.

[24]  J. Henner,et al.  The personal is political in The Deaf Mute Howls: deaf epistemology seeks disability justice , 2017 .

[25]  Veton Kepuska,et al.  Comparing Speech Recognition Systems (Microsoft API, Google API And CMU Sphinx) , 2017 .

[26]  Larwan Berke,et al.  Displaying confidence from imperfect automatic speech recognition for captioning , 2017, ACM SIGACCESS Access. Comput..

[27]  Richard E. Ladner,et al.  Improving Real-Time Captioning Experiences for Deaf and Hard of Hearing Students , 2016, ASSETS.

[28]  Vicki L. Hanson,et al.  SlidePacer: A Presentation Delivery Tool for Instructors of Deaf and Hard of Hearing Students , 2016, ASSETS.

[29]  Matt Huenerfauth,et al.  Deaf and Hard of Hearing Individuals' Perceptions of Communication with Hearing Colleagues in Small Groups , 2016, ASSETS.

[30]  Kelly Caine,et al.  Local Standards for Sample Size at CHI , 2016, CHI.

[31]  Raja S. Kushalnagar,et al.  Tracked Speech-To-Text Display: Enhancing Accessibility and Readability of Real-Time Speech-To-Text , 2015, ASSETS.

[32]  Brent N. Shiver,et al.  Evaluating Alternatives for Better Deaf Accessibility to Selected Web-Based Multimedia , 2015, ASSETS.

[33]  Denis Jouvet,et al.  Qualitative investigation of the display of speech recognition results for communication with deaf people , 2015, SLPAT@Interspeech.

[34]  Matthew Louis Mauriello,et al.  Understanding the Role of Thermography in Energy Auditing: Current Practices and the Potential for Automated Solutions , 2015, CHI.

[35]  Jon Froehlich,et al.  Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing , 2015, CHI.

[36]  Gill Harold,et al.  DeafSpace and the principles of universal design , 2014, Disability and rehabilitation.

[37]  Poorna Kushalnagar,et al.  Collaborative Gaze Cues and Replay for Deaf and Hard of Hearing Students , 2014, ICCHP.

[38]  Walter S. Lasecki,et al.  Helping students keep up with real-time captions by pausing and highlighting , 2014, W4A.

[39]  Gregg C. Vanderheiden,et al.  Crowd caption correction (CCC) , 2013, ASSETS.

[40]  John Mourjopoulos,et al.  Automatic speech recognition performance in different room acoustic environments with and without dereverberation preprocessing , 2013, Comput. Speech Lang..

[41]  Walter S. Lasecki,et al.  Online quality control for real-time crowd captioning , 2012, ASSETS '12.

[42]  Walter S. Lasecki,et al.  A readability evaluation of real-time crowd captions in the classroom , 2012, ASSETS '12.

[43]  Eric C. Larson,et al.  The design and evaluation of prototype eco-feedback displays for fixture-level water usage data , 2012, CHI.

[44]  Marco Furini,et al.  Enhancing learning accessibility through fully automatic captioning , 2012, W4A.

[45]  Meng Wang,et al.  Dynamic captioning: video accessibility enhancement for hearing impairment , 2010, ACM Multimedia.

[46]  Gillian R. Hayes,et al.  Disability studies as a source of critical inquiry for the field of assistive technology , 2010, ASSETS '10.

[47]  Richard E. Ladner,et al.  ClassInFocus: enabling improved visual attention strategies for deaf and hard of hearing students , 2009, Assets '09.

[48]  Thomas Way,et al.  Inclusion of deaf students in computer science classes using real-time speech transcription , 2007, ITiCSE '07.

[49]  P. David Stotts,et al.  Semi-transparent video interfaces to assist deaf persons in meetings , 2007, ACM-SE 45.

[50]  R. Mitchell,et al.  How Many People Use ASL in the United States? Why Estimates Need Updating , 2006 .

[51]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[52]  M. Wald.  Using Automatic Speech Recognition to Enhance Education for All Students: Turning a Vision into Reality , 2005, Proceedings Frontiers in Education 35th Annual Conference.

[53]  H. Lane.  Ethnicity, ethics, and the deaf-world. , 2005, Journal of deaf studies and deaf education.

[54]  H-Dirksen L. Bauman.  Audism: exploring the metaphysics of oppression. , 2004, Journal of deaf studies and deaf education.

[55]  M. Maclagan,et al.  Speaking rates of American and New Zealand varieties of English , 2004 .

[56]  M. Maclagan,et al.  Speaking rates of American and New Zealand varieties of English. , 2001, Clinical linguistics & phonetics.

[57]  Allison Druin,et al.  Technology probes: inspiring design for and with families , 2003, CHI '03.

[58]  Sara H. Basson,et al.  Speech recognition in university classrooms: liberated learning project , 2002, Assets '02.

[59]  D. Haggerty,et al.  What do we mean by a , 2001 .

[60]  D. Gorenflo,et al.  Preventive attitudes and beliefs of deaf and hard-of-hearing individuals. , 2000, Archives of family medicine.

[61]  Carl Jensema.  Viewer Reaction to Different Television Captioning Speeds , 1998, American annals of the deaf.

[62]  Michael Oliver,et al.  Social Work with Disabled People , 1983 .

[63]  U. Bellugi,et al.  A comparison of sign language and spoken language , 1972 .