Auditory Interfaces and Sonification

Auditory interfaces and sonifi cation—information display by means of nonspeech audio (Kramer et al., 1999)—have been the subject of increasing interest in recent decades (for reviews, see Kramer et al., 1999; Frysinger, 2005). With the advent of ubiquitous digital technologies, high-fi delity sound samples have become increasingly easy and inexpensive to produce and implement (Hereford and Winn, 1994; Flowers, Buhman, and Turnage, 2005). Perhaps more important, however, an increasing awareness of the shortcomings and limitations of traditional visual interfaces has spurred research on sound as a viable mode of information display. Nonspeech audio cues have been implemented to varying degrees in interface design, ranging from nonspeech audio as a complement or supplement to existing visual displays (e.g., Brown, Newsome, and Glinert, 1989; Brewster, 1997), to hybrid systems that integrate nonspeech audio with other audio technologies (e.g., screen readers; see Morley et al., 1999; Stockman, Hind, and Frauenberger, 2005). Attempts have even been made to develop interfaces (usually for the visually impaired) where feedback and interaction are driven primarily by sounds (e.g., Bonebright and Nees, in press; Edwards, 1989a, 1989b; Mynatt, 1997). Despite the potential utility of sound in interface design, a recent survey of experts in HCI and usability (Frauenberger, Stockman, and Bourguet, 2007a) reported that only about 58% of respondents had designed with audio in any form. Nonspeech audio and sonifi cation represent an important tool for universally accessible interface design, yet most interface designers consider speech audio fi rst (and perhaps exclusively) when implementing audio in a system. Perhaps as a relic of the limited sound production capabilities of early personal computers (see Flowers, Buhman, and Turnage, 2005), perceptions (and in some cases legitimate concerns) linger that sounds in interfaces are a minimally informative annoyance to the user. ! is chapter argues that appropriately chosen and implemented nonspeech sounds can be a pleasant, informative, and integral part of interface design, and interfaces with nonspeech audio can promote adherence to at least fi ve of the seven principles of universal design (Connell et al., 1997; McGuire, Scott, and Shaw, 2006), including (1) equitable use; (2) fl exibility in use; (3) simple and intuitive use; (4) perceptible information; and (5) tolerance for error. ! e current chapter seeks to provide an introduction to nonspeech auditory information display and an overview of the relevant issues and critical decision points regarding the use of nonspeech audio in interfaces. ! e discussion is guided by the theme that nonspeech auditory displays can universally enhance the human operator’s experience with human-machine systems. As this chapter focuses on the potential benefi ts of nonspeech audio, the interested reader is referred to other chapters in this volume (e.g., Chapter 28, “Screen Readers,” Chapter 30, “Speech Input to Support Universal Access,” and Chapter 40, “Contributions of ‘Ambient’ Multimodality to Universal Access”) for a complete discussion of the range of interface options available to the auditory or multimodal display engineer. 32

[1]  Bruce N. Walker,et al.  Musical soundscapes for an Accessible Aquarium: Bringing Dynamic Exhibits to the Visually Impaired , 2007, ICMC.

[2]  W. Turnbull,et al.  Pitch discrimination as a function of tonal duration , 1944 .

[3]  Bruce N. Walker,et al.  Advanced Auditory Menus , 2007 .

[4]  John P. S. Salmen Everyone's Welcome: The Americans with Disabilities Act and Museums. , 1998 .

[5]  Bruce N. Walker,et al.  SONIFICATION SANDBOX: A GRAPHICAL TOOLKIT FOR AUDITORY GRAPHS , 2003 .

[6]  Meera Blattner,et al.  Earcons and Icons: Their Structure and Common Design Principles , 1989, Hum. Comput. Interact..

[7]  Lorna M. Brown,et al.  DRAWING BY EAR: INTERPRETING SONIFIED LINE GRAPHS , 2003 .

[8]  Jonathan C. Roberts,et al.  A path based model for sonification , 2004 .

[9]  Bruce N. Walker,et al.  AQUARIUM SONIFICATION: SOUNDSCAPES FOR ACCESSIBLE DYNAMIC INFORMAL LEARNING ENVIRONMENTS , 2006 .

[10]  Bruce N. Walker,et al.  Mappings and metaphors in auditory displays: An experimental assessment , 2005, TAP.

[11]  A Postma,et al.  Interactions between Exogenous Auditory and Visual Spatial Attention , 2000, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[12]  Mark A. Ericson,et al.  Factors That Influence Intelligibility in Multitalker Speech Displays , 2004 .

[13]  Bruce N. Walker,et al.  PSYCHOPHYSICAL SCALING OF SONIFICATION MAPPINGS: A COMPARISION OF VISUALLY IMPAIRED AND SIGHTED LISTENERS , 2001 .

[14]  Robert S. Bolia,et al.  Aurally Aided Visual Search in Three-Dimensional Space , 1999, Hum. Factors.

[15]  Judy Edworthy,et al.  Designing trend-monitoring sounds for helicopters: methodological issues and an application. , 2004, Journal of experimental psychology. Applied.

[16]  Raymond M. Stanley,et al.  Lateralization of Sounds Using Bone-Conduction Headsets , 2006 .

[17]  Stephen A. Brewster,et al.  An evaluation of earcons for use in auditory human-computer interfaces , 1993, INTERCHI.

[18]  Bruce N. Walker,et al.  An agenda for research and development of multimodal graphs , 2005 .

[19]  David M. Lane,et al.  SONIFICATION OF ABSOLUTE VALUES WITH SINGLE AND MULTIPLE DIMENSIONS , 2003 .

[20]  Thomas Hermann,et al.  An introduction to interactive sonification , 2005 .

[21]  F L Wightman,et al.  Headphone simulation of free-field listening. II: Psychophysical validation. , 1989, The Journal of the Acoustical Society of America.

[22]  John H. Flowers,et al.  Cross-Modal Equivalence of Visual and Auditory Scatterplots for Exploring Bivariate Data Samples , 1997, Hum. Factors.

[23]  James R. Marston,et al.  Personal Guidance System for People with Visual Impairment: A Comparison of Spatial Displays for Route Guidance , 2005, Journal of visual impairment & blindness.

[24]  Bruce N. Walker,et al.  Ecological Psychoacoustics and Auditory Displays: Hearing, Grouping, and Meaning Making , 2004 .

[25]  Alistair D. N. Edwards,et al.  Modelling Blind Users' Interactions with an Auditory Computer Interface , 1989, Int. J. Man Mach. Stud..

[26]  Nigel Papworth,et al.  BEOWULF: A GAME EXPERIENCE BUILT ON SOUND EFFECTS , 2007 .

[27]  Alistair D. N. Edwards,et al.  Soundtrack: An Auditory Interface for Blind Users (Abstract Only) , 1989, SGCH.

[28]  M. Quinn,et al.  RESEARCH SET TO MUSIC: THE CLIMATE SYMPHONY AND OTHER SONIFICATIONS OF ICE CORE, RADAR, DNA, SEISMIC AND SOLAR WIND DATA , 2001 .

[29]  Bruce N. Walker,et al.  Individual Differences, Cognitive Abilities, and the Interpretation of Auditory Graphs , 2004, ICAD.

[30]  Christopher Frauenberger,et al.  Pattern Design in the Context Space A Methodological Framework for Auditory Display Design , 2007 .

[31]  Benjamin K. Davison,et al.  Sonification Sandbox Reconstruction: Software Standard for Auditory Graphs , 2007 .

[32]  Stephen Brewster,et al.  Using non-speech sound to overcome information overload , 1997 .

[33]  S. Resnikoff,et al.  Global data on visual impairment in the year 2002. , 2004, Bulletin of the World Health Organization.

[34]  Bruce N. Walker,et al.  Data density and trend reversals in auditory graphs: Effects on point-estimation and trend-identification tasks , 2008, TAP.

[35]  Weijia Zhou,et al.  Proceedings of the Human Factors and Ergonomics Society , 1996 .

[36]  Judy Edworthy Does sound help us to work better with machines? A commentary on Rauterberg's paper ‘About the importance of auditory alarms during the operation of a plant simulator’ , 1998 .

[37]  Wai Yu,et al.  Using Non-speech Sounds to Improve Access to 2D Tabular Numerical Information for Visually Impaired Users , 2001, BCS HCI/IHM.

[38]  Tony Stockman,et al.  The design of interactive audio soccer , 2007 .

[39]  Helen Petrie,et al.  Auditory navigation in hyperspace: Design and evaluation of a non-visual hypermedia system for blind users , 1999, Behav. Inf. Technol..

[40]  S. Camille Peres,et al.  Auditory graphs: The effects of redundant dimensions and divided attention , 2005 .

[41]  Bruce N. Walker,et al.  Creating Functional and Livable Soundscapes for Peripheral Monitoring of Dynamic Data , 2004, ICAD.

[42]  Fredrik Winberg,et al.  QUALITATIVE ASPECTS OF AUDITORY DIRECT MANIPULATION A CASE STUDY OF THE TOWERS OF HANOI , 2001 .

[43]  Christopher Frauenberger,et al.  INTERACTIVE SONIFICATION OF SPREADSHEETS , 2005 .

[44]  Frank Dellaert,et al.  SWAN: System for Wearable Audio Navigation , 2007, 2007 11th IEEE International Symposium on Wearable Computers.

[45]  Bruce N. Walker,et al.  Consistency of magnitude estimations with conceptual data dimensions used for sonification , 2007 .

[46]  Stephen Barrass,et al.  Auditory information design , 1998 .

[47]  Christopher Frauenberger,et al.  A survey on common practice in designing audio in the user interface , 2007, BCS HCI.

[48]  Elsa Eiriksdottir,et al.  User Preferences for Auditory Device-driven Menu Navigation , 2006 .

[49]  Perry R. Cook,et al.  Din of an ``Iniquity'': Analysis and Synthesis of Environmental Sounds , 2007 .

[50]  Jonathan Grudin,et al.  Partitioning digital worlds: focal and peripheral awareness in multiple monitor use , 2001, CHI.

[51]  John H. Flowers,et al.  Data sonification from the desktop: Should sound be part of standard data analysis software? , 2005, TAP.

[52]  S. Hillyard,et al.  Involuntary orienting to sound improves visual perception , 2000, Nature.

[53]  Stephen A. Brewster,et al.  Understanding concurrent earcons: Applying auditory scene analysis principles to concurrent earcon recognition , 2004, TAP.

[54]  Bruce N Walker,et al.  Magnitude estimation of conceptual data dimensions for use in sonification. , 2002, Journal of experimental psychology. Applied.

[55]  John H. Flowers,et al.  Musical versus Visual Graphs: Cross-Modal Equivalence in Perception of Time Series Data , 1995, Hum. Factors.

[56]  Martin Eimer,et al.  Crossmodal links in spatial attention between vision, audition, and touch: evidence from event-related brain potentials , 2001, Neuropsychologia.

[57]  Bruce N. Walker,et al.  Navigation Performance With a Virtual Auditory Display: Effects of Beacon Sound, Capture Radius, and Practice , 2006, Hum. Factors.

[58]  Bruce N. Walker,et al.  The Effect of a Speech Discrimination Task on Navigation in a Virtual Environment , 2006 .

[59]  William Winn,et al.  Non-Speech Sound in Human-Computer Interaction: A Review and Design Guidelines , 1994 .

[60]  Bruce N. Walker,et al.  Effects of Auditory Context Cues and Training on Performance of a Point Estimation Sonification Task , 2005 .

[61]  Mikael Fernström,et al.  Audio games: Fun for all? All for fun! , 2003 .

[62]  S. S. Stevens A scale for the measurement of a psychological magnitude: loudness. , 1936 .

[63]  Ephraim P. Glinert,et al.  An experiment into the use of auditory cues to reduce visual workload , 1989, CHI '89.

[64]  Elizabeth D. Mynatt,et al.  Mapping GUIs to auditory interfaces , 1992, UIST '92.

[65]  Judy Edworthy,et al.  Designing urgency into auditory warnings using pitch, speed and loudness , 1996 .

[66]  Jim Tobias Information Technology and Universal Design: An Agenda for Accessible Technology , 2003 .

[67]  Elizabeth D. Mynatt Transforming Graphical Interfaces Into Auditory Interfaces for Blind Users , 1997, Hum. Comput. Interact..

[68]  Alberto de Campo,et al.  Toward a data sonification design space map , 2007 .

[69]  Fergus R. Fricke,et al.  Duration of Perceived and Performed Sounds , 1997 .

[70]  Bruce N. Walker,et al.  BRIEF TRAINING FOR PERFORMANCE OF A POINT ESTIMATION SONIFICATION TASK , 2005 .

[71]  Raymond M. Stanley,et al.  THRESHOLDS OF AUDIBILITY FOR BONE-CONDUCTION HEADSETS , 2005 .

[72]  Gregory Kramer,et al.  An introduction to auditory display , 1994 .

[73]  Bruce N. Walker,et al.  Listener, Task, and Auditory Graph: Toward a Conceptual Model of Auditory Graph Comprehension , 2007 .

[74]  John H. Flowers,et al.  THIRTEEN YEARS OF REFLECTION ON AUDITORY GRAPHING: PROMISES, PITFALLS, AND POTENTIAL NEW DIRECTIONS , 2005 .

[75]  Lorna M. Brown,et al.  BROWSING MODES FOR EXPLORING SONIFIED LINE GRAPHS , 2002 .

[76]  Susan Anable,et al.  Accessibility Techniques for Museum Web Sites. , 2001 .

[77]  Schloss Birlinghoven,et al.  USING AUDIFICATION IN PLANETARY SEISMOLOGY , 2001 .

[78]  Xiao-Li Yang,et al.  Designing a personal guidance system to aid navigation without sight: progress on the GIS component , 1991, Int. J. Geogr. Inf. Sci..

[79]  William W. Gaver The SonicFinder: An Interface That Uses Auditory Icons , 1989, Hum. Comput. Interact..

[80]  Hesham M. Kamel,et al.  A Comparison of Three Nonvisual Methods for Presenting Scientific Graphs , 2002 .

[81]  Jennifer Mankoff,et al.  Is your web page accessible?: a comparative study of methods for assessing web page accessibility for the blind , 2005, CHI.

[82]  Steven M. Hetzler,et al.  TWO TOOLS FOR INTEGRATING SONIFICATION INTO CALCULUS INSTRUCTION 1 , 2006 .

[83]  D Griffith,et al.  Computer Access for Persons Who Are Blind or Visually Impaired: Human Factors Issues , 1990, Human factors.

[84]  C. Spence,et al.  Exogenous spatial cuing studies of human crossmodal attention and multisensory integration , 2004 .

[85]  Bruce N Walker,et al.  Using Virtual Environments to Prototype Auditory Navigation Displays , 2005, Assistive technology : the official journal of RESNA.

[86]  Lorna M. Brown,et al.  DESIGN GUIDELINES FOR AUDIO PRESENTATION OF GRAPHS AND TABLES , 2003 .

[87]  M. Boltz,et al.  Tempo discrimination of musical patterns: Effects due to pitch and rhythmic structure , 1998, Perception & psychophysics.

[88]  Albert S. Bregman,et al.  The Auditory Scene. (Book Reviews: Auditory Scene Analysis. The Perceptual Organization of Sound.) , 1990 .

[89]  E. B. Newman,et al.  A Scale for the Measurement of the Psychological Magnitude Pitch , 1937 .

[90]  John H. Flowers,et al.  The ear’s versus the eye’s potential to assess characteristics of numeric data: Are we too visuocentric? , 1992 .

[91]  Anne Meyer,et al.  Universal Design for Individual Differences. , 2000 .

[92]  Bruce N. Walker,et al.  Human Factors and the Acoustic Ecology: Considerations for Multimedia Audio Design , 1996 .

[93]  Emmanuel Bigand,et al.  Contributions of music to research on human auditory cognition. , 1993 .

[94]  Penelope M. Sanderson,et al.  Overlapping Melodic Alarms Are Almost Indiscriminable , 2007, Hum. Factors.

[95]  B. Shinn-Cunningham,et al.  Note on informational masking. , 2003, The Journal of the Acoustical Society of America.

[96]  Fredrik Winberg,et al.  Designing accessible auditory drag and drop , 2002, CUU '03.

[97]  Joan M. McGuire,et al.  Universal Design and Its Applications in Educational Environments , 2006 .

[98]  Steven P. Frysinger,et al.  A Brief History of Auditory Data Representation to the 1980s , 2005 .

[99]  John H. Flowers,et al.  “Sound” alternatives to visual graphics for exploratory data analysis , 1993 .

[100]  Charles S. Watson,et al.  Factors in the design of effective auditory displays , 1994 .

[101]  T A Mondor,et al.  Effect of same- and different-modality spatial cues on auditory and visual target identification. , 1998, Journal of experimental psychology. Human perception and performance.

[102]  Dennis J. Folds The Elevation Illusion in Virtual Audio , 2006 .

[103]  Elaine Gerber,et al.  The Benefits of and Barriers to Computer use for Individuals who are Visually Impaired , 2003 .

[104]  L. Mauney Individual Differences in Cognitive, Musical, and Perceptual Abilities , 2006 .

[105]  S. Camille Peres,et al.  Sonification of statistical graphs , 2003 .

[106]  Terri L. Bonebright,et al.  Sonific ation Report: Status of the Field and Research Agenda , 2010 .

[107]  Derek Brock,et al.  Using an auditory display to manage attention in a dual task, multiscreen environment , 2002 .

[108]  Gregory Kramer,et al.  Pitch and loudness interact in auditory displays: can the data get lost in the map? , 2002, Journal of experimental psychology. Applied.

[109]  Bruce N. Walker,et al.  LEARNING RATES FOR AUDITORY MENUS ENHANCED WITH SPEARCONS VERSUS EARCONS , 2007 .

[110]  Edward Childs Auditory graphs of real-time data , 2005 .

[111]  Michael A. Nees,et al.  Memory for Auditory Icons and Earcons with Localization Cues , 2007 .

[112]  Roberta L. Klatzky,et al.  Personal guidance system for the visually impaired , 1994, ASSETS.

[113]  F L Wightman,et al.  Headphone simulation of free-field listening. I: Stimulus synthesis. , 1989, The Journal of the Acoustical Society of America.

[114]  Peter Keller,et al.  Meaning from environmental sounds: types of signal-referent relations and their effect on recognizing auditory icons. , 2004, Journal of experimental psychology. Applied.

[115]  William W. Gaver,et al.  Effective sounds in complex systems: the ARKOLA simulation , 1991, CHI.

[116]  Stephen A. Brewster,et al.  Overcoming the Lack of Screen Space on Mobile Computers , 2002, Personal and Ubiquitous Computing.

[117]  B. Shinn-Cunningham,et al.  Note on informational masking (L) , 2003 .

[118]  Bruce N. Walker,et al.  The Audio Abacus: Representing a Wide Range of Values with Accuracy and Precision , 2004, ICAD.

[119]  Bruce N. Walker,et al.  SPEARCONS: SPEECH-BASED EARCONS IMPROVE NAVIGATION PERFORMANCE IN AUDITORY MENUS , 2006 .

[120]  Terri L. Bonebright,et al.  Testing the effectiveness of sonified graphs for education: A programmatic research project , 2001 .