HoloSound: Combining Speech and Sound Identification for Deaf or Hard of Hearing Users on a Head-mounted Display

Head-mounted displays can provide private and glanceable speech and sound feedback to deaf and hard of hearing people, yet prior systems have largely focused on speech transcription. We introduce HoloSound, a HoloLens-based augmented reality (AR) prototype that uses deep learning to classify and visualize sound identity and location in addition to providing speech transcription. This poster paper presents a working proof-of-concept prototype, and discusses future opportunities for advancing AR-based sound awareness.

[1]  Jon E. Froehlich,et al.  Autoethnography of a Hard of Hearing Traveler , 2019, ASSETS.

[2]  Walter S. Lasecki,et al.  Accessibility Evaluation of Classroom Captions , 2014, ACM Trans. Access. Comput..

[3]  Jon Froehlich,et al.  SoundWatch: Exploring Smartwatch-based Deep Learning Approaches to Support Sound Awareness for Deaf and Hard of Hearing Users , 2020, ASSETS.

[4]  Angela Lin,et al.  Exploring Sound Awareness in the Home for People who are Deaf or Hard of Hearing , 2019, CHI.

[5]  Jon Froehlich,et al.  Towards Accessible Conversations in a Mobile Context for People who are Deaf and Hard of Hearing , 2018, ASSETS.

[6]  Jon Froehlich,et al.  Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing , 2015, CHI.

[7]  Kelly Mack,et al.  HomeSound: An Iterative Field Deployment of an In-Home Sound Awareness System for Deaf or Hard of Hearing Users , 2020, CHI.

[8]  Raja S. Kushalnagar,et al.  Deaf, Hard of Hearing, and Hearing Perspectives on Using Automatic Speech Recognition in Conversation , 2017, ASSETS.

[9]  Jon Froehlich,et al.  Exploring Augmented Reality Approaches to Real-Time Captioning: A Preliminary Autoethnographic Study , 2018, Conference on Designing Interactive Systems.

[10]  Jon Froehlich,et al.  Evaluating Smartwatch-based Sound Feedback for Deaf and Hard-of-hearing Users Across Contexts , 2020, CHI.

[11]  P. van Kranenburg,et al.  International Society for Music Information Retrieval , 2014 .

[12]  Richard E. Ladner,et al.  A Personalizable Mobile Sound Detector App Design for Deaf and Hard-of-Hearing Users , 2016, ASSETS.

[13]  François Michaud,et al.  Lightweight and Optimized Sound Source Localization and Tracking Methods for Open and Closed Microphone Array Configurations , 2018, Robotics Auton. Syst..

[14]  Jon Froehlich,et al.  Deaf and Hard-of-hearing Individuals' Preferences for Wearable and Mobile Sound Awareness Technologies , 2019, CHI.

[15]  Xavier Serra,et al.  Freesound Datasets: A Platform for the Creation of Open Audio Datasets , 2017, ISMIR.

[16]  Benjamin M. Gorman.  VisAural:: a wearable sound-localisation device for people with impaired hearing , 2014, ASSETS.

[17]  L. Bernstein,et al.  Temporal and spatio-temporal vibrotactile displays for voice fundamental frequency: an initial evaluation of a new vibrotactile speech perception aid with normal-hearing and hearing-impaired individuals. , 1998, The Journal of the Acoustical Society of America.

[18]  Aren Jansen,et al.  CNN architectures for large-scale audio classification , 2016, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[19]  Mike Y. Chen,et al.  SpeechBubbles: Enhancing Captioning Experiences for Deaf and Hard-of-Hearing People in Group Conversations , 2018, CHI.