Exploration of Automatic Speech Recognition for Deaf and Hard of Hearing Students in Higher Education Classes

Automatic speech recognition (ASR) programs that generate real-time speech-to-text captions can be provided as supplemental access technologies for deaf and hard of hearing (DHH) students in higher education classes. As part of a pilot program, we implemented ASR as a supplemental access service in biology, statistics, and other courses at our university. To identify the benefits and limitations of ASR as an access technology, we surveyed 26 DHH students and interviewed 8 of these students about their experiences with ASR in their mainstream classes. Participants believed that ASR was beneficial despite the errors that ASR continued to generate; however, the accuracy and readability of ASR need to improve so that students can better access spoken information through ASR. This paper reviews points for researchers to consider when designing and providing ASR as a supplemental access service in educational settings.

[1]  Walter S. Lasecki,et al.  Accessibility Evaluation of Classroom Captions , 2014, ACM Trans. Access. Comput..

[2]  Matt Huenerfauth,et al.  Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing , 2017, ASSETS.

[3]  Walter S. Lasecki,et al.  A readability evaluation of real-time crowd captions in the classroom , 2012, ASSETS '12.

[4]  Jan Nouza,et al.  Speaker-adaptive speech recognition using speaker diarization for improved transcription of large spoken archives , 2013, Speech Commun..

[5]  Gerald Penn,et al.  Automatic human utility evaluation of ASR systems: does WER really predict performance? , 2013, INTERSPEECH.

[6]  Raja S. Kushalnagar,et al.  Tracked Speech-To-Text Display: Enhancing Accessibility and Readability of Real-Time Speech-To-Text , 2015, ASSETS.

[7]  Seiichi Nakagawa,et al.  Improving the Readability of ASR Results for Lectures Using Multiple Hypotheses and Sentence-Level Knowledge , 2012, IEICE Trans. Inf. Syst..

[8]  Thomas Way,et al.  Inclusion of deaf students in computer science classes using real-time speech transcription , 2007, ITiCSE '07.

[9]  Larwan Berke,et al.  Deaf and Hard-of-Hearing Perspectives on Imperfect Automatic Speech Recognition for Captioning One-on-One Meetings , 2017, ASSETS.

[10]  Jungyun Seo,et al.  Post-error Correction in Automatic Speech Recognition Using Discourse Information , 2014 .

[11]  Yashesh Gaur,et al.  The effects of automatic speech recognition quality on human transcription latency , 2016, W4A.

[12]  Marco Furini,et al.  Enhancing learning accessibility through fully automatic captioning , 2012, W4A.

[13]  Hynek Hermansky,et al.  Predicting error rates for unknown data in automatic speech recognition , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[14]  Harmanpreet Kaur,et al.  Towards More Robust Speech Interactions for Deaf and Hard of Hearing Users , 2018, ASSETS.

[15]  Dilek Z. Hakkani-Tür,et al.  Active learning: theory and applications to automatic speech recognition , 2005, IEEE Transactions on Speech and Audio Processing.

[16]  Richard E. Ladner,et al.  Improving Real-Time Captioning Experiences for Deaf and Hard of Hearing Students , 2016, ASSETS.

[17]  Denis Jouvet,et al.  Qualitative investigation of the display of speech recognition results for communication with deaf people , 2015, SLPAT@Interspeech.

[18]  M. Stinson,et al.  College Students' Perceptions of the C-Print Speech-to-Text Transcription System. , 2001, Journal of deaf studies and deaf education.

[19]  Mike Wald,et al.  Universal access to communication and learning: the role of automatic speech recognition , 2008, Universal Access in the Information Society.

[20]  Mike Wald.  Captioning for Deaf and Hard of Hearing People by Editing Automatic Speech Recognition in Real Time , 2006, ICCHP.

[21]  Horia Cucu,et al.  Word error rate improvement and complexity reduction in Automatic Speech Recognition by analyzing acoustic model uncertainty and confusion , 2011, 2011 6th Conference on Speech Technology and Human-Computer Dialogue (SpeD).

[22]  Sara H. Basson,et al.  Speech recognition in university classrooms: liberated learning project , 2002, Assets '02.

[23]  Alfred Mertins,et al.  Automatic speech recognition and speech variability: A review , 2007, Speech Commun..