Evaluation of the effects of spatial separation and timbral differences on the identifiability of features of concurrent auditory streams

When using non-speech as an information delivery medium, information patterns are mapped onto auditory variables. The efficiency of delivering information to listeners depends on the usage of auditory variables. A discriminable auditory display can reduce the ambiguity of listeners’ perceptual representation of auditory inputs. However, there is lacking an understanding of the way in which listeners decode information embedded in sound through identifying auditory variations. This thesis aims to develop an understanding of the influence of auditory variables on the identifiability of the features of auditory display. The auditory display in this thesis relates to auditory graph where series of quantitative values were mapped onto pitches and presented over time. This thesis explores the effects of spatial separation and timbre on the identifiability of the features of simultaneously presented auditory graphs. Features that listeners were required to identify included an increased periodic modulation in pitch that occurred in only one stream of two displayed auditory streams or a difference in the overall shape of the pitch contour of two displayed auditory streams. Both of these tasks required divided attention between stimuli forming two auditory streams, which were presented via headphones in order either to appear at two separate spatial locations, or to be co-located at a single location. Also, the displayed stimuli could differ in timbre, allowing for an evaluation of the relative influences of timbral difference and spatial separation on the identifiability of the target features. One hypothesis tested here was that when simultaneously presented auditory graphs were composed of the same timbre, their pitch contours would be more easily identified when displayed at spatially separated locations. A second hypothesis tested here was that the enhancement of identification performance due to spatial separation of simultaneously presented auditory graphs would be greatest for auditory streams that differ least in timbre, and that discrimination performance given streams with more distinct timbre would be less influenced by spatial separation of the streams. In a subsequent divided attention study using only spatially separated auditory streams, the perceptual salience of timbral differences in an identification task was investigated using timbral attribute ratings. Finally, in a data sonification case study employing multiple auditory streams exhibiting spatial and timbral differences, a qualitative approach was taken that aimed to deepen understanding of the way people listen and think in such contexts. Overall, the applied method demonstrated the influence of spatial separation and timbral differences on the feature identification for simultaneously presented auditory streams. Although there is potential for further refinement of the methods applied here, this work provided an examination and elucidation of the relative salience of two prominent factors involved in determining the effectiveness of concurrent auditory displays and delineated some of their limitations in supporting two different identification tasks. The responses, collected using an open questionnaire administered after participation in some of the experiments, highlighted the intricate relationships that exist in the allocation of attention between perceptually segregated concurrent streams, and in the auditory display processing activities and listening strategies of human subjects engaged in divided attention tasks.

[1]  G. E. Peterson,et al.  Control Methods Used in a Study of the Vowels , 1951 .

[2]  E. C. Cherry Some Experiments on the Recognition of Speech, with One and with Two Ears , 1953 .

[3]  C. Osgood,et al.  The Measurement of Meaning , 1958 .

[4]  L. N. Solomon Semantic Approach to the Perception of Complex Sounds , 1958 .

[5]  Heinrich Hertz,et al.  On the differences between localization and lateralization. , 1974, The Journal of the Acoustical Society of America.

[6]  G. Von Bismarck,et al.  Sharpness as an attribute of the timbre of steady sounds , 1974 .

[7]  E. Terhardt On the perception of periodic sound fluctuations (roughness) , 1974 .

[8]  P. E. Doak,et al.  A subjective rating scale for timbre , 1976 .

[9]  B. Moore An Introduction to the Psychology of Hearing , 1977 .

[10]  Thomas J. Moore Voice communications jamming research , 1981 .

[11]  Michaël Titus Maria Scheffers,et al.  Sifting vowels. Auditory pitch analysis and sound segregation. , 1983 .

[12]  R. Yin Case Study Research: Design and Methods , 1984 .

[13]  B. Moore,et al.  Thresholds for the detection of inharmonicity in complex tones. , 1985, The Journal of the Acoustical Society of America.

[14]  Harold L. Hawkins,et al.  Auditory information processing. , 1986 .

[15]  Chris Marshall,et al.  Design guidelines , 1987 .

[16]  F L Wightman,et al.  Headphone simulation of free-field listening. I: Stimulus synthesis. , 1989, The Journal of the Acoustical Society of America.

[17]  A. Bregman,et al.  The perceptual segregation of simultaneous auditory signals: Pulse train segregation and vowel segregation , 1989, Perception & psychophysics.

[18]  A J Houtsma,et al.  Pitch identification of simultaneous diotic and dichotic two-tone complexes. , 1989, The Journal of the Acoustical Society of America.

[19]  William W. Gaver The SonicFinder: An Interface That Uses Auditory Icons , 1989, Hum. Comput. Interact..

[20]  Hugo Fastl,et al.  Psychoacoustics: Facts and Models , 1990 .

[21]  Meera M. Blattner,et al.  Communicating and Learning Through Non-speech Audio , 1992 .

[22]  J. Culling,et al.  Auditory segregation of competing voices: absence of effects of FM or AM coherence. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[23]  Stephen McAdams,et al.  Recognition of sound sources and events , 1993 .

[24]  Charles Darwin,et al.  Perceptual separation of concurrent vowels: within and across-formant grouping by F0 , 1993 .

[25]  F L Wightman,et al.  Localization using nonindividualized head-related transfer functions. , 1993, The Journal of the Acoustical Society of America.

[26]  E. Carterette,et al.  Verbal Attributes of Simultaneous Wind Instrument Timbres: II. Adjectives Induced from Piston's "Orchestration" , 1993 .

[27]  William W. Gaver How Do We Hear in the World?: Explorations in Ecological Acoustics , 1993 .

[28]  Jean-Marc Jot,et al.  Spat~ : A Spatial Processor for Musicians and Sound Engineers , 1995 .

[29]  J. Culling,et al.  Perceptual separation of concurrent speech sounds: absence of across-frequency grouping by common interaural delay. , 1995, The Journal of the Acoustical Society of America.

[30]  Jean-Marc Jot,et al.  Digital Signal Processing Issues in the Context of Binaural and Transaural Stereophony , 1995 .

[31]  G. Soete,et al.  Perceptual scaling of synthesized musical timbres: Common dimensions, specificities, and latent subject classes , 1995, Psychological research.

[32]  P. Iverson,et al.  Auditory stream segregation by musical timbre: effects of static and dynamic acoustic attributes. , 1995, Journal of experimental psychology. Human perception and performance.

[33]  Elena Samoylenko,et al.  Systematic Analysis of Verbalizations Produced in Comparing Musical Timbres , 1996 .

[34]  Ajm Adrian Houtsma,et al.  Pitch and timbre : definition, meaning and use , 1997 .

[35]  Christophe Hourdin,et al.  A Multidimensional Scaling Analysis of Musical Instruments' Time-Varying Spectra , 1997 .

[36]  Hideki Kawahara,et al.  Concurrent vowel identification. I. Effects of relative amplitude and F0 difference , 1997, The Journal of the Acoustical Society of America.

[37]  R. W. Hukin,et al.  Auditory objects of attention: the role of interaural time differences. , 1999, Journal of experimental psychology. Human perception and performance.

[38]  F L Wightman,et al.  Resolution of front-back ambiguity in spatial hearing by listener and source movement. , 1999, The Journal of the Acoustical Society of America.

[39]  N. Chouard,et al.  A semantic differential design especially developed for the evaluation of interior car sounds , 1999 .

[40]  V. Ralph Algazi,et al.  An adaptable ellipsoidal head model for the interaural time difference , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[41]  J. H. Frey,et al.  The Interview: From Structured Questions to Negotiated Text , 2000 .

[42]  Vincent Rioux Sound Quality of Flue Organ Pipe. An Interdisciplinary Study on the Art of Voicing , 2001 .

[43]  William L. Martens,et al.  Multidimensional Perceptual Scaling of Tone Color Variation in Three Modeled Guitar Amplifiers , 2002 .

[44]  D S Brungart,et al.  Informational and energetic masking effects in the perception of two simultaneous talkers. , 2001, The Journal of the Acoustical Society of America.

[45]  John G. Neuhoff,et al.  PITCH CHANGE, SONIFICATION, AND MUSICAL EXPERTISE: WHICH WAY IS UP? , 2002 .

[46]  Lorna M. Brown,et al.  BROWSING MODES FOR EXPLORING SONIFIED LINE GRAPHS , 2002 .

[47]  B. Kapralos,et al.  Auditory Perception and Spatial Auditory Systems4 , 2002 .

[48]  Douglas S Brungart,et al.  The effects of spatial separation in distance on the informational and energetic masking of a nearby speech signal. , 2002, The Journal of the Acoustical Society of America.

[49]  G. Kidd,et al.  Similarity, uncertainty, and masking in the identification of nonspeech auditory patterns. , 2002, The Journal of the Acoustical Society of America.

[50]  Bruce N. Walker,et al.  Tick-marks, axes, and labels: The effects of adding context to auditory graphs , 2002 .

[51]  André van Schaik,et al.  TWO-POINT DISCRIMINATION IN AUDITORY DISPLAYS , 2003 .

[52]  Lorna M. Brown,et al.  DRAWING BY EAR: INTERPRETING SONIFIED LINE GRAPHS , 2003 .

[53]  André van Schaik,et al.  7TWO-POINT DISCRIMINATION IN AUDITORY DISPLAYS , 2003 .

[54]  Min-Shik Kim,et al.  The role of spatial working memory in visual search efficiency , 2004, Psychonomic bulletin & review.

[55]  Mark A. Ericson,et al.  Factors That Influence Intelligibility in Multitalker Speech Displays , 2004 .

[56]  Bruce N. Walker,et al.  Individual Differences, Cognitive Abilities, and the Interpretation of Auditory Graphs , 2004, ICAD.

[57]  Kenneth I. Joy,et al.  Sound graphs: A numerical data analysis method for the blind , 1985, Journal of Medical Systems.

[58]  Barbara G. Shinn-Cunningham,et al.  Selective and Divided Attention: Extracting Information from Simultaneous Sound Sources , 2004, ICAD.

[59]  Stephen A. Brewster,et al.  Space, the Final Frontearcon: The Identification of Concurrently Presented Earcons in a Synthetic Spatialised Auditory Environment , 2004, ICAD.

[60]  Fabio C. Ciardi sMax: A multimodal toolkit for stock market data sonification , 2004 .

[61]  G. Woodman,et al.  Visual search is slowed when visuospatial working memory is occupied , 2004, Psychonomic bulletin & review.

[62]  Hani Yehia,et al.  Timbre Classification Of A Single Musical Instrument , 2004, ISMIR.

[63]  B. Weedon,et al.  Echoic Memory In Primitive Auditory Selective Attention , 2004 .

[64]  Bruce N. Walker,et al.  BRIEF TRAINING FOR PERFORMANCE OF A POINT ESTIMATION SONIFICATION TASK , 2005 .

[65]  Frederick J. Gallun,et al.  The advantage of knowing where to listen. , 2005, The Journal of the Acoustical Society of America.

[66]  S. Jan,et al.  Verbal description of musical sound timbre in Czech language and its relation to musicians profession and performance quality , 2005 .

[67]  Paul Vickers WHITHER AND WHEREFORE THE AUDITORY GRAPH? ABSTRACTIONS & ÆSTHETICS IN AUDITORY AND SONIFIED GRAPHS , 2005 .

[68]  Lisa M. Mauney INDIVIDUAL DIFFERENCES IN INTERPRETING AUDITORY GRAPHS , 2005 .

[69]  Hong Jun Song,et al.  Time-based Sonification for Information Representation , 2005 .

[70]  Diemo Schwarz,et al.  Ftm - Complex Data Structures for Max , 2005, ICMC.

[71]  Gil Weinberg,et al.  Interactive Sonification: Aesthetics, Functionality and Performance , 2006, Leonardo Music Journal.

[72]  Densil Cabrera,et al.  An interactive approach for teaching information sonification , 2006 .

[73]  J. Stepánek MUSICAL SOUND TIMBRE: VERBAL DESCRIPTION AND DIMENSIONS , 2006 .

[74]  David McGookin,et al.  ADVANTAGES AND ISSUES WITH CONCURRENT AUDIO PRESENTATION AS PART OF AN AUDITORY DISPLAY , 2006 .

[75]  A. Baddeley,et al.  The multi-component model of working memory: Explorations in experimental cognitive psychology , 2006, Neuroscience.

[76]  Ka Beilharz,et al.  Concurrent Auditory Stream Discrimination in Auditory Graphing , 2007 .

[77]  Densil Cabrera,et al.  Evaluation of Spatial Presentation in Sonification for Identifying Concurrent Audio Streams , 2007 .

[78]  Kirsty A. Beilharz,et al.  Spatialization and timbre for effective auditory graphing , 2007 .

[79]  Kirsty A. Beilharz,et al.  Aesthetic and auditory enhancements for multi-stream information sonification , 2008, DIMEA.

[80]  A. Bregman Auditory Scene Analysis , 2008 .

[81]  William A. Yost Auditory, Localization and Scene Perception , 2008 .

[82]  B. Shinn-Cunningham Object-based auditory and visual attention , 2008, Trends in Cognitive Sciences.

[83]  Udo Zoelzer,et al.  DAFX: Digital Audio Effects , 2011 .

[84]  Cécile Colin,et al.  The effect of spatial separation on informational masking: Presentation of an original dichotic paradigm , 2012 .