Audemes at work: Investigating features of non-speech sounds to maximize content recognition

To access interactive systems, blind users can leverage their auditory senses by using non-speech sounds. The structure of existing non-speech sounds, however, is geared toward conveying atomic operations at the user interface (e.g., opening a file) rather than evoking broader, theme-based content typical of educational material (e.g., an historical event). To address this problem, we investigate audemes, a new category of non-speech sounds whose semiotic structure and flexibility open new horizons for the aural interaction with content-rich applications. Three experiments with blind participants examined the attributes of an audeme that most facilitate the accurate recognition of their meaning. A sequential concatenation of different sound types (music, sound effect) yielded the highest meaning recognition, whereas an overlapping arrangement of sounds of the same type (music, music) yielded the lowest meaning recognition. We discuss seven guidelines to design well-formed audemes.

[1]  David Sonnenschein,et al.  Sound Design: The Expressive Power of Music, Voice and Sound Effects in Cinema , 2001 .

[2]  Eoin Brazil,et al.  Human-Computer Interaction Design based on Interactive Sonification - Hearing Actions or Instruments/Agents. , 2004 .

[4]  Sarah Guri-Rozenblit,et al.  The interrelations between diagrammatic representations and verbal explanations in learning from social science texts , 1988 .

[5]  Stephen Brewster,et al.  Experimentally Derived Guidelines for the Creation of Earcons , 2001 .

[6]  Mexhid Ferati,et al.  Educational Sound Symbols for the Visually Impaired , 2009, HCI.

[7]  Mark S. Sanders,et al.  Human Factors in Engineering and Design , 2016 .

[8]  A. Paivio,et al.  Dual coding theory and education , 1991 .

[9]  A. Baddeley.  Essentials of Human Memory , 1999 .

[10]  William W. Gaver.  Auditory Icons: Using Sound in Computer Interfaces , 1986, Hum. Comput. Interact..

[11]  Barry H. Kantowitz,et al.  Human Factors: Understanding People-System Relationships , 1983 .

[12]  Antti Pirhonen,et al.  NON-SPEECH SOUNDS AS ELEMENTS OF A USE SCENARIO: A SEMIOTIC PERSPECTIVE , 2006 .

[13]  J. Laird,et al.  Remembering What You Feel: Effects of Emotion on Memory , 1982 .

[14]  William W. Gaver.  The SonicFinder: An Interface That Uses Auditory Icons , 1989, Hum. Comput. Interact..

[15]  Stephen A. Brewster,et al.  Designing Sound: Towards a System for Designing Audio Interfaces using Timbre Spaces , 2004, ICAD.

[16]  Paul A Lucas,et al.  An evaluation of the communicative ability of auditory icons and earcons , 1994 .

[17]  Mexhid Ferati,et al.  Usability evaluation of acoustic interfaces for the blind , 2011, SIGDOC '11.

[18]  Myounghoon Jeon,et al.  “Spindex”: Accelerated Initial Speech Sounds Improve Navigation Performance in Auditory Menus , 2009 .

[19]  Helen Petrie,et al.  Auditory navigation in hyperspace: design and evaluation of a non-visual hypermedia system for blind users , 1998, Assets '98.

[20]  Stephen Brewster,et al.  Providing an audio glance at algebra for blind readers , 1994 .

[21]  R. Mayer,et al.  Nine Ways to Reduce Cognitive Load in Multimedia Learning , 2003 .

[22]  Stephen A. Brewster,et al.  Earcons as a Method of Providing Navigational Cues in a Menu Hierarchy , 1996, BCS HCI.

[23]  G. Bower.  Mood and memory. , 1981, The American psychologist.

[24]  Jaime Sánchez,et al.  Interactive virtual acoustic environments for blind children: computing, usability, and cognition , 2001, CHI Extended Abstracts.

[25]  Maria Klara Wolters,et al.  Name that tune: musicons as reminders in the home , 2011, CHI.

[26]  Michel Chion,et al.  Audio-Vision: Sound on Screen , 1994 .

[27]  Stéphane Conversy.  Ad-hoc synthesis of auditory icons , 1998 .

[28]  Stephen Brewster,et al.  Providing a structured method for integrating Non-Speech Audio into HCI , 1994 .

[29]  Mexhid Ferati,et al.  Using Audemes as a Learning Medium for the Visually Impaired , 2009, HEALTHINF.

[30]  Mexhid Ferati,et al.  Acoustic interaction design through "audemes": experiences with the blind , 2009, SIGDOC '09.

[31]  A. D. N. Edwards,et al.  Weasel: a computer based system for providing non-visual access to music notation , 2000, SIGC.

[32]  Catherine Plaisant,et al.  Non-visual exploration of geographic maps: Does sonification help? , 2010, Disability and rehabilitation. Assistive technology.

[33]  Stephen A. Brewster,et al.  The design of sonically-enhanced widgets , 1998, Interact. Comput..

[34]  Alistair D. N. Edwards,et al.  Soundtrack: An Auditory Interface for Blind Users (Abstract Only) , 1989, SGCH.

[35]  Marco Fabiani,et al.  Interactive sonification of emotionally expressive gestures by means of music performance , 2010 .

[36]  Bruce N. Walker,et al.  SPEARCONS: SPEECH-BASED EARCONS IMPROVE NAVIGATION PERFORMANCE IN AUDITORY MENUS , 2006 .

[37]  Hideki Koike,et al.  EdgeSonic: image feature sonification for the visually impaired , 2011, AH '11.

[38]  Robert David Stevens,et al.  Principles for the Design of Auditory Interfaces to Present Complex Information to Blind People , 1996 .

[39]  Jaime Sánchez,et al.  Memory enhancement through audio , 2004, ACM SIGACCESS Access. Comput..

[40]  Lisa Feldman Barrett,et al.  Sex Differences in Emotional Awareness , 2000 .

[41]  Rebecca L. Oxford,et al.  Instructional Implications of Gender Differences in Second/Foreign Language (L2) Learning Styles and Strategies. , 1993 .

[42]  Tilman Dingler,et al.  Learnabiltiy of Sound Cues for Environmental Features: Auditory Icons, Earcons, Spearcons, and Speech , 2008 .

[43]  Antti Pirhonen,et al.  Same sound – Different meanings : A Novel Scheme for Modes of Listening , 2010 .

[44]  Seunghun Kim,et al.  Composition With Path : Musical Sonification Of Geo- Referenced Data With Online Map Interface , 2010, ICMC.

[45]  M. Lassonde,et al.  Blind subjects process auditory spectral cues more efficiently than sighted individuals , 2004, Experimental Brain Research.

[46]  Stephen A. Brewster,et al.  Parallel earcons: reducing the length of audio messages , 1995, Int. J. Hum. Comput. Stud..

[47]  Catherine Guastavino,et al.  USABILITY OF NON-SPEECH SOUNDS IN USER INTERFACES , 2008 .

[48]  E. Klima.  The signs of language , 1979 .

[49]  Elizabeth D. Mynatt.  Designing with auditory icons: how well do we identify auditory cues? , 1994, CHI Conference Companion.

[50]  R. Mayer,et al.  A Split-Attention Effect in Multimedia Learning: Evidence for Dual Processing Systems in Working Memory , 1998 .

[51]  William W. Gaver,et al.  Effective sounds in complex systems: the ARKOLA simulation , 1991, CHI.

[52]  E. Diener,et al.  Sex differences in the recall of affective experiences. , 1998, Journal of personality and social psychology.

[53]  Stephen A. Brewster,et al.  An evaluation of earcons for use in auditory human-computer interfaces , 1993, INTERCHI.

[54]  Robert M. Bernard.  Using extended captions to improve learning from instructional illustrations , 1990, Br. J. Educ. Technol..

[55]  M. McDaniel,et al.  Illustrations as adjuncts to prose: a text-appropriate processing approach , 1988 .

[56]  Jaime Sánchez,et al.  AudioMath: Blind children learning mathematics through audio , 2005 .

[57]  Eli Hagen,et al.  Towards an American Sign Language interface , 1994, Artificial Intelligence Review.

[58]  A. Glenberg,et al.  Comprehension of illustrated text: Pictures help to build mental models☆ , 1992 .

[59]  Manne-Sakari Mustonen,et al.  A review-based conceptual analysis of auditory signs and their design , 2008 .

[60]  Maribeth Back,et al.  Micro-narratives in sound design: Context, character, and caricature in waveform manipulation , 1996 .

[61]  Stephen M. Kosslyn,et al.  Elements of graph design , 1993 .

[62]  Meera Blattner,et al.  Earcons and Icons: Their Structure and Common Design Principles , 1989, Hum. Comput. Interact..