Identifying Major Components of Pictures by Audio Encoding of Colours

The goal of the See ColOr project is to achieve a non-invasive mobility aid for blind users that will use the auditory pathway to represent in real-time frontal image scenes. More particularly, we have developed a prototype which transforms HSL coloured pixels into spatialized classical instrument sounds lasting for 300 ms. Hue is sonified by the timbre of a musical instrument, saturation is one of four possible notes, and luminosity is represented by bass when luminosity is rather dark and singing voice when it is relatively bright. Our first experiments are devoted to static images on the computer screen. Six participants with their eyes covered by a dark tissue were trained to associate colours with musical instruments and then asked to determine on several pictures, objects with specific shapes and colours. In order to simplify the protocol of experiments, we used a tactile tablet, which took the place of the camera. Overall, experiment participants found that colour was helpful for the interpretation of image scenes.

[1]  J. Cronly-Dillon,et al.  The perception of visual images encoded in musical form: a study in cross-modality information transfer , 1999, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[2]  José L. González-Mora,et al.  Development of a New Space Perception System for Blind People, Based on the Creation of a Virtual Acoustic Space , 1999, IWANN.

[3]  Vincent Danos,et al.  Self Assembling Graphs , 2005, IWINAC.

[4]  L. Kay,et al.  A sonar aid to enhance spatial perception of the blind: engineering design and evaluation , 1974 .

[5]  Peter B. L. Meijer,et al.  An experimental system for auditory image representations , 1992, IEEE Transactions on Biomedical Engineering.

[6]  Guido Bologna,et al.  Eye Tracking in Coloured Image Scenes Represented by Ambisonic Fields of Musical Instrument Sounds , 2005, IWINAC.

[7]  R M Ruff,et al.  Auditory spatial pattern perception aided by visual choices , 1976, Psychological research.

[8]  José Mira,et al.  Engineering Applications of Bio-Inspired Artificial Neural Networks , 1999, Lecture Notes in Computer Science.

[9]  David G. Malham,et al.  3-D Sound Spatialization using Ambisonic Techniques , 1995 .

[10]  C. Trullemans,et al.  A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution , 1998, IEEE Transactions on Biomedical Engineering.

[11]  Michael A. Gerzon,et al.  Psychoacoustic Decoders for Multispeaker Stereo and Surround Sound , 1992 .

[12]  S. Lakatos.  Recognition of Complex Auditory-Spatial Patterns , 1993, Perception.

[13]  C. Avendano,et al.  The CIPIC HRTF database , 2001, Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat. No.01TH8575).