Curve shape and curvature perception through interactive sonification

In this article we present an approach that uses sound to communicate geometrical data related to a virtual object. This has been developed in the framework of a multimodal interface for product design. The interface allows a designer to evaluate the quality of a 3-D shape using touch, vision, and sound. Two important considerations addressed in this article are the nature of the data that is sonified and the haptic interaction between the user and the interface, which in fact triggers the sound and influences its characteristics. Based on these considerations, we present a number of sonification strategies that are designed to map the geometrical data of interest into sound. The fundamental frequency of various sounds was used to convey the curve shape or the curvature to the listeners. Two evaluation experiments are described, one involves partipants with a varied background, the other involved the intended users, i.e. participants with a background in industrial design. The results show that independent of the sonification method used and independent of whether the curve shape or the curvature were sonified, the sonification was quite successful. In the first experiment participants had a success rate of about 80% in a multiple choice task, in the second experiment it took the participants on average less than 20 seconds to find the maximum, minimum or inflection points of the curvature of a test curve.

[1]  Eric Moulines,et al.  Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones , 1989, Speech Commun..

[2]  Tony Stockman,et al.  AUDITORY GRAPHS: A SUMMARY OF CURRENT EXPERIENCE AND TOWARD S A RESEARCH AGENDA , 2005 .

[3]  Hesham M. Kamel,et al.  A Comparison of Three Nonvisual Methods for Presenting Scientific Graphs , 2002 .

[4]  Bruce N. Walker,et al.  Consistency of magnitude estimations with conceptual data dimensions used for sonification , 2007 .

[5]  S. Lacey,et al.  Vision and Touch: Multiple or Multisensory Representations of Objects? , 2007, Perception.

[6]  Stephen Barrass,et al.  Auditory information design , 1998 .

[7]  Andrea R. Kennel.  Audiograf: a diagram-reader for the blind , 1996, Assets '96.

[8]  G. Rizzolatti,et al.  The mirror-neuron system. , 2004, Annual review of neuroscience.

[9]  M. Leman.  Embodied Music Cognition and Mediation Technology , 2007 .

[10]  J. Mezrich,et al.  Dynamic Representation of Multivariate Time Series Data , 1984 .

[11]  Daniel Smilek,et al.  Geometric Shape Detection with Soundview , 2004, ICAD.

[12]  Suresh K. Lodha,et al.  Musart: Musical audio transfer function real-time toolkit , 2002 .

[13]  Bruce N Walker,et al.  Magnitude estimation of conceptual data dimensions for use in sonification. , 2002, Journal of experimental psychology. Applied.

[14]  Dj Dik Hermes,et al.  Perception of rubbing sounds , 2008 .

[15]  Dennis Roseman,et al.  What Should a Surface in 4-Space Look Like? , 1997, VisMath.

[16]  Ronen Barzel,et al.  Audio Anecdotes II: Tools, Tips, and Techniques for Digital Audio , 2004 .

[17]  Patrick Roth,et al.  Graphics and User's Exploration via Simple Sonics (GUESS): Providing Interrelational Representation of Objects in a Non-visual Environment , 2001 .

[18]  S Lakatos,et al.  Haptic form perception: Relative salience of local and global features , 1999, Perception & psychophysics.

[19]  Thomas Hermann,et al.  Sonification for Exploratory Data Analysis , 2002 .

[20]  Alistair D. N. Edwards,et al.  Access to Mathematics for Visually Disabled Students Through Multimodal Interaction , 1997, Hum. Comput. Interact..

[21]  Dj Dik Hermes,et al.  Perception of real rubbing sounds , 2009 .

[22]  Michael Pettitt,et al.  Interactive Sonification of Curve Shape and Curvature Data , 2009, HAID.

[23]  Petr Janata,et al.  Marketbuzz: Sonification of Real-Time Financial Data , 2004, ICAD.

[24]  Kenneth I. Joy,et al.  Sound graphs: A numerical data analysis method for the blind , 1985, Journal of Medical Systems.

[25]  M. Leman,et al.  Musical gestures : sound, movement, and meaning , 2010 .

[26]  S. Shelley,et al.  Evaluating geometrical properties of virtual shapes using interactive sonification , 2008, 2008 IEEE International Workshop on Haptic Audio visual Environments and Games.

[27]  D. M. Green,et al.  Frequency discrimination as a function of frequency and sensation level. , 1977, The Journal of the Acoustical Society of America.

[28]  Bruce N. Walker,et al.  Creating Functional and Livable Soundscapes for Peripheral Monitoring of Dynamic Data , 2004, ICAD.

[29]  G. Rizzolatti,et al.  Hearing Sounds, Understanding Actions: Action Representation in Mirror Neurons , 2002, Science.

[30]  Monica Bordegoni,et al.  Haptic and Sound Interface for Shape Rendering , 2010, PRESENCE: Teleoperators and Virtual Environments.

[31]  Joachim Gossmann,et al.  Towards an auditory representation of complexity , 2005 .

[32]  Monica Bordegoni,et al.  Geodesic Spline Interface for Haptic Curve Rendering , 2011, IEEE Transactions on Haptics.

[33]  Hans-Christian Hege,et al.  Visualization and mathematics - experiments, simulations and environments , 1997 .

[34]  Suresh K. Lodha,et al.  MUSE: A musical data sonification toolkit , 1997 .

[35]  R. Daniel Bergeron,et al.  Stereophonic and surface sound generation for exploratory data analysis , 1990, CHI '90.

[36]  Petr Janata,et al.  Marketbuzz: Sonification of real-time financial dataa , 2004 .

[37]  Suresh K. Lodha,et al.  Listen: A data sonification toolkit , 1996 .

[38]  Bruce N. Walker,et al.  Mappings and metaphors in auditory displays: An experimental assessment , 2005, TAP.

[39]  Insook Choi,et al.  Investigating geometric data with sound , 1996 .

[40]  Hans-Christian Hege,et al.  Visualization and Mathematics III , 2011 .

[41]  Bruce N. Walker,et al.  SONIFICATION SANDBOX: A GRAPHICAL TOOLKIT FOR AUDITORY GRAPHS , 2003 .

[42]  Jeff Sauro,et al.  Average task times in usability tests: what to report? , 2010, CHI 2010.

[43]  Neville H Fletcher,et al.  Principles of Vibration and Sound , 1994 .

[44]  Marina Basu.  The Embodied Mind: Cognitive Science and Human Experience , 2004 .

[45]  Franca Giannini,et al.  A Survey of Computer-Aided Modeling Tools for Aesthetic Design , 2002, J. Comput. Inf. Sci. Eng..

[46]  K.V. Nesbitt,et al.  Finding trading patterns in stock market data , 2004, IEEE Computer Graphics and Applications.

[47]  T. Furness,et al.  Perception of virtual auditory shapes , 1994 .

[48]  G. Von Bismarck,et al.  Sharpness as an attribute of the timbre of steady sounds , 1974 .

[49]  Hideko F. Norman,et al.  The visual and haptic perception of natural object shape , 2004, Perception & Psychophysics.

[50]  Snehashish Chakraverty,et al.  Vibration of Plates , 2008 .

[51]  Rosane Minghim,et al.  An illustrated analysis of sonification for scientific visualisation , 1995, Proceedings Visualization '95.

[52]  Gregory Kramer,et al.  Auditory Display: Sonification, Audification, And Auditory Interfaces , 1994 .

[53]  Herbert Edelsbrunner,et al.  Auditory Morse Analysis of Triangulated Manifolds , 1997, VisMath.

[54]  Isabella Poggi,et al.  Gestures in performance , 2009 .