A Taxonomy of Sounds in Virtual Reality

Virtual reality (VR) leverages human sight, hearing and touch senses to convey virtual experiences. For d/Deaf and hard of hearing (DHH) people, information conveyed through sound may not be accessible. To help with future design of accessible VR sound representations for DHH users, this paper contributes a consistent language and structure for representing sounds in VR. Using two studies, we report on the design and evaluation of a novel taxonomy for VR sounds. Study 1 included interviews with 10 VR sound designers to develop our taxonomy along two dimensions: sound source and intent. To evaluate this taxonomy, we conducted another study (Study 2) where eight HCI researchers used our taxonomy to document sounds in 33 VR apps. We found that our taxonomy was able to successfully categorize nearly all sounds (265/267) in these apps. We also uncovered additional insights for designing accessible visual and haptic-based sound substitutes for DHH users.

[1]  Kathrin Maria Gerling,et al.  A Critical Examination of Virtual Reality Technology in the Context of the Minority Body , 2021, CHI.

[2]  Jon Froehlich,et al.  SoundWatch: Exploring Smartwatch-based Deep Learning Approaches to Support Sound Awareness for Deaf and Hard of Hearing Users , 2020, ASSETS.

[3]  Edward Cutrell,et al.  Virtual Reality Without Vision: A Haptic and Auditory White Cane to Navigate Complex Virtual Worlds , 2020, CHI.

[4]  Kelly Mack,et al.  HomeSound: An Iterative Field Deployment of an In-Home Sound Awareness System for Deaf or Hard of Hearing Users , 2020, CHI.

[5]  Ting Zhang,et al.  Identifying Comfort Areas in 3D Space for Persons with Upper Extremity Mobility Impairments Using Virtual Reality , 2019, ASSETS.

[6]  Meredith Ringel Morris,et al.  Accessible by Design: An Opportunity for Virtual Reality , 2019, 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct).

[7]  Jon Froehlich,et al.  Deaf and Hard-of-hearing Individuals' Preferences for Wearable and Mobile Sound Awareness Technologies , 2019, CHI.

[8]  Kyle Rector,et al.  Virtual Showdown: An Accessible Virtual Reality Game with Scaffolds for Youth with Visual Impairments , 2019, CHI.

[9]  David Philip Green,et al.  Behind the Curtain of the "Ultimate Empathy Machine": On the Composition of Virtual Reality Nonfiction Experiences , 2019, CHI.

[10]  Angela Lin,et al.  Exploring Sound Awareness in the Home for People who are Deaf or Hard of Hearing , 2019, CHI.

[11]  Meredith Ringel Morris,et al.  SeeingVR: A Set of Tools to Make Virtual Reality More Accessible to People with Low Vision , 2019, CHI.

[12]  Franz Zotter,et al.  Ambisonics: A Practical 3D Audio Theory for Recording, Studio Production, Sound Reinforcement, and Virtual Reality , 2019 .

[13]  Jon Froehlich,et al.  Towards Accessible Conversations in a Mobile Context for People who are Deaf and Hard of Hearing , 2018, ASSETS.

[14]  Mauro Teófilo,et al.  Exploring Virtual Reality to Enable Deaf or Hard of Hearing Accessibility in Live Theaters: A Case Study , 2018, HCI.

[15]  Jon Froehlich,et al.  Exploring Augmented Reality Approaches to Real-Time Captioning: A Preliminary Autoethnographic Study , 2018, Conference on Designing Interactive Systems.

[16]  Mike Y. Chen,et al.  SpeechBubbles: Enhancing Captioning Experiences for Deaf and Hard-of-Hearing People in Group Conversations , 2018, CHI.

[17]  Meredith Ringel Morris,et al.  Enabling People with Visual Impairments to Navigate Virtual Reality with a Haptic and Auditory Cane Simulation , 2018, CHI.

[18]  Meredith Ringel Morris,et al.  Rich Representations of Visual Content for Screen Reader Users , 2018, CHI.

[19]  David J. Chalmers,et al.  The Virtual and the Real , 2017 .

[20]  Aren Jansen,et al.  Audio Set: An ontology and human-labeled dataset for audio events , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[21]  Ines Kozuh,et al.  Community building among deaf and hard of hearing people by using written language on social networking sites , 2016, Comput. Hum. Behav..

[22]  Richard E. Ladner,et al.  A Personalizable Mobile Sound Detector App Design for Deaf and Hard-of-Hearing Users , 2016, ASSETS.

[23]  PerMagnus Lindborg,et al.  A taxonomy of sound sources in restaurants , 2016 .

[24]  Yvonne Jaeger.  Film Sound Theory And Practice , 2016 .

[25]  A. Zajadacz,et al.  Evolution of models of disability as a basis for further policy changes in accessible tourism , 2015 .

[26]  Jon Froehlich,et al.  Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing , 2015, CHI.

[27]  Justin Salamon,et al.  A Dataset and Taxonomy for Urban Sound Research , 2014, ACM Multimedia.

[28]  Sigal Eden,et al.  Enhancing Storytelling Ability with Virtual Environment among Deaf and Hard-of-Hearing Children , 2014, ICCHP.

[29]  Benjamin Wright.  Footsteps with character: the art and craft of Foley , 2014 .

[30]  Harlan Lane,et al.  Deaf Ethnicity, Deafhood, and Their Relationship , 2013 .

[31]  Bo-sun Xie,et al.  Head-Related Transfer Function and Virtual Auditory Display: E-Book , 2013 .

[32]  Jacob O. Wobbrock,et al.  In the shadow of misperception: assistive technology use and social interactions , 2011, CHI.

[33]  Krzysztof Z. Gajos,et al.  Ability-Based Design: Concept, Principles and Examples , 2011, TACC.

[34]  Kristine Jørgensen.  Time for New Terminology? Diegetic and Non-Diegetic Sounds in Computer Games Revisited , 2010 .

[35]  L. Christidis,et al.  Systematics And Taxonomy Of Australian Birds , 2008 .

[36]  Richard E. Ladner,et al.  Hearing Impairments , 2008, Web Accessibility.

[37]  Wendell Hanna.  The New Bloom's Taxonomy: Implications for Music Education , 2007 .

[38]  Tara Matthews,et al.  Evaluating non-speech sound visualizations for the deaf , 2006, Behav. Inf. Technol..

[39]  Surendra Ranganath,et al.  Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Astrid Paeschke,et al.  A database of German emotional speech , 2005, INTERSPEECH.

[41]  Alexander Brandon.  Audio for Games: Planning, Process, and Production (New Riders Games) , 2004 .

[42]  Dan Gärdenfors,et al.  Audio games: new perspectives on game audio , 2004, ACE '04.

[43]  M. Scherer,et al.  Assistive Technology Use and Stigma , 2004 .

[44]  A. Stockburger,et al.  The game environment from an auditive perspective , 2003 .

[45]  Sigal Eden,et al.  Virtual Reality as a Tool for Improving Spatial Rotation among Deaf and Hard-of-Hearing Children , 2001, Cyberpsychology Behav. Soc. Netw..

[46]  D.A. Nelson,et al.  Database design for an archive of animal sounds , 2001, IEEE Engineering in Medicine and Biology Magazine.

[47]  Marti L. Riemer-Reiss,et al.  Factors Associated with Assistive Technology Discontinuance among Individuals with Disabilities , 2000 .

[48]  François Pachet,et al.  A taxonomy of musical genres , 2000, RIAO.

[49]  Hitoshi Iida,et al.  A speech and language database for speech translation research , 1994, ICSLP.

[50]  Matthew S. Moore,et al.  For hearing people only: Answers to some of the most commonly asked questions about the Deaf community, its culture, and the "Deaf reality" , 1992 .

[51]  A. Strauss,et al.  Grounded theory , 2017 .

[52]  Jock D. Mackinlay,et al.  The design space of input devices , 1990, CHI '90.