Environmental Sounds as Concept Carriers for Communication

Sonification, the use of nonspeech audio to represent data and information, has been applied to industrial systems and computer interfaces via mechanisms such as auditory icons and earcons. In this paper, we explore a different application of sonification, which is to facilitate communication across language barriers by conveying commonly used concepts via environmental auditory representations. SoundNet, a linguistic database enhanced with natural nonspeech audio, is constructed for this purpose. The concept-sound associations which are building blocks of SoundNet were validated through a sound labeling study conducted on Amazon Mechanical Turk. We determine the factors that cause a sound to evoke a concept. We examine which aspects of the proposed auditory representations are evocative, and what kinds of confusions may occur. Our results show that sounds can effectively illustrate some concepts, especially those related to concrete entities and actions, and thus can be utilized in assistive communication applications.

[1]  Elizabeth D. Mynatt.  Designing with auditory icons: how well do we identify auditory cues? , 1994, CHI Conference Companion.

[2]  John G. Neuhoff,et al.  Sonification Report: Status of the Field and Research Agenda Prepared for the National Science Foundation by members of the International Community for Auditory Display , 1999 .

[3]  Durand R. Begault,et al.  A Virtual Audio Guidance and Alert System for Commercial Aircraft Operations , 1996 .

[4]  William W. Gaver,et al.  Effective sounds in complex systems: the ARKOLA simulation , 1991, CHI.

[5]  S. Handel,et al.  Listening: An Introduction to the Perception of Auditory Events , 1993 .

[6]  A. D. Groot,et al.  Conceptual representation in bilingual memory: Effects of concreteness and cognate status in word association , 1998, Bilingualism: Language and Cognition.

[7]  G. Miller,et al.  A Semantic Network of English Verbs , 1998 .

[8]  Perry R. Cook,et al.  How well do visual verbs work in daily communication for young and old adults? , 2009, CHI.

[9]  William W. Gaver.  The SonicFinder: An Interface That Uses Auditory Icons , 1989, Hum. Comput. Interact..

[10]  William W. Gaver.  What in the World Do We Hear? An Ecological Approach to Auditory Event Perception , 1993 .

[11]  D. Norman,et al.  Everyday listening and auditory icons , 1988 .

[12]  S. Backhaus,et al.  Quantum oscillations between two weakly coupled reservoirs of superfluid 3He , 1997, Nature.

[13]  George Tzanetakis,et al.  Musical genre classification of audio signals , 2002, IEEE Trans. Speech Audio Process..

[14]  Perry R. Cook,et al.  PERCEPTUAL SPACES FOR SOUND EFFECTS OBTAINED WITH AN INTERACTIVE SIMILARITY RATING PROGRAM , 2001 .

[15]  E. Owens,et al.  An Introduction to the Psychology of Hearing , 1997 .

[16]  M. Sliwinski,et al.  Causal Uncertainty in the Identification of Environmental Sounds , 1986 .

[17]  Tomoya Takasaki.  PictNet: Semantic Infrastructure for Pictogram Communication , 2006 .

[18]  Eamonn O'Neill,et al.  Auditory icon and earcon mobile service notifications: intuitiveness, learnability, memorability and preference , 2009, CHI.

[19]  Stephanie Clarke,et al.  Non-verbal auditory recognition in normal subjects and brain-damaged patients: Evidence for parallel processing , 1996, Neuropsychologia.

[20]  Stephen A. Brewster,et al.  Using nonspeech sounds to provide navigation cues , 1998, TCHI.

[21]  Meera Blattner,et al.  Earcons and Icons: Their Structure and Common Design Principles , 1989, Hum. Comput. Interact..

[22]  Perry R. Cook,et al.  Speaking through pictures: images vs. icons , 2009, Assets '09.

[23]  R. Shaw,et al.  Persistence and Change : Proceedings of the First International Conference on Event Perception , 1985 .

[24]  J. Ballas.  Common factors in the identification of an assortment of brief everyday sounds , 1993 .

[25]  Terri L. Bonebright,et al.  Sonific ation Report: Status of the Field and Research Agenda , 2010 .

[26]  Ayse Pinar Saygin,et al.  The effects of linguistic mediation on the identification of environmental sounds , 2002 .