SOUNDVIEW: SENSING COLOR IMAGES BY KINESTHETIC AUDIO

An experimental system called SoundView has been developed, which allows the exploration of a color image through touch and hearing. The image is mapped onto a virtual surface with a ne-grained color dependent roughness texture. The user explores the image by moving a pointer device over the image. The pointer acts like a virtual gramophone needle, and the sound produced depends on the motion as well as on the color of the area explored. The roughness texture is obtained by constructing a mapping of three dimensional color space onto a three dimensional sound space. The mapping tries to achieve maximal alignment of the color and sound spaces by preserving the perceptual metrical and topological structure of color space, as well as by incorporating common associations between sound and color.

[1]  Vilayanur S. Ramachandran,et al.  Acquired synesthesia in retinitis pigmentosa , 1999 .

[2]  Matthew Rizzo,et al.  Synesthesia , 1989, Neurology.

[3]  Gail Levy-Berger,et al.  Human Experimental Psychology , 1985 .

[4]  Dinesh K. Pai,et al.  Perception of Material from Contact Sounds , 2000, Presence: Teleoperators & Virtual Environments.

[5]  Dinesh K. Pai,et al.  MEASUREMENTS OF PERCEPTUAL QUALITY OF CONTACT SOUND MODELS , 2002 .

[6]  R. Shepard.  Circularity in Judgments of Relative Pitch , 1964 .

[7]  Helen Petrie,et al.  Use of a haptic device by blind and sighted people : perception of virtual textures and objects , 1998 .

[8]  Kenneth Steiglitz,et al.  A digital signal processing primer - with applications to digital audio and computer music , 1996 .

[9]  Calle Sjöström,et al.  Designing haptic computer interfaces for blind people , 2001, ISSPA.

[10]  Dinesh K. Pai,et al.  JASS: A JAVA AUDIO SYNTHESIS SYSTEM FOR PROGRAMMERS , 2001 .

[11]  Kenneth E. Barner,et al.  HAPTIC REPRESENTATION OF SCIENTIFIC DATA FOR VISUALLY IMPAIRED OR BLIND PERSONS , 1996 .

[12]  Stephen Brewster,et al.  Feeling Rough: Multimodal Perception of Virtual Roughness , 2001 .

[13]  Peter B. L. Meijer,et al.  An experimental system for auditory image representations , 1992, IEEE Transactions on Biomedical Engineering.

[14]  William W. Gaver.  Synthesizing auditory icons , 1993, INTERCHI.

[15]  Eric Krotkov,et al.  Robotic Perception of Material: Experiments with Shape-Invariant Acoustic Measures of Material Type , 1995, ISER.

[16]  Stephen A. Brewster,et al.  Constructing sonified haptic line graphs for the blind student: first steps , 2000, Assets '00.

[17]  I. Placencia Porrero,et al.  Improving the Quality of Life for the European Citizen: Technology for Inclusive Design and Equality , 1998 .

[18]  Leslie Kay.  A CTFM acoustic spatial sensing technology: its use by blind persons and robots , 1999 .