The Vibe: a versatile vision-to-audition sensory substitution device

We describe a sensory substitution scheme that converts a video stream into an audio stream in real-time. It was initially developed as a research tool for studying human ability to learn new ways of perceiving the world: the Vibe can give us the ability to learn a kind of 'vision' by audition. It converts a video stream into a continuous stereophonic audio signal that conveys information coded from the video stream. The conversion from the video stream to the audio stream uses a kind of retina with receptive fields. Each receptive field controls a sound source and the user listens to a sound that is a mixture of all these sound sources. Compared to other existing vision-to-audition sensory substitution devices, the Vibe is highly versatile in particular because it uses a set of configurable units working in parallel. In order to demonstrate the validity and interest of this method of vision to audition conversion, we give the results of an experiment involving a pointing task to targets memorised through visual perception or through their auditory conversion by the Vibe. This article is also an opportunity to precisely draw the general specifications of this scheme in order to prepare its implementation on an autonomous/mobile hardware.

[1]  BENJAMIN WHITE,et al.  Vision Substitution by Tactile Image Projection , 1969, Nature.

[2]  R Bruyer,et al.  The Ponzo Illusion with Auditory Substitution of Vision in Sighted and Early-Blind Subjects , 2005, Perception.

[3]  F. Lacquaniti,et al.  Short-Term Memory for Reaching to Visual Targets: Psychophysical Evidence for Body-Centered Reference Frames , 1998, The Journal of Neuroscience.

[4]  Jeanny Hérault,et al.  Biomimetic Space-Variant Sampling in a Vision Prosthesis Improves the User's Skill in a Localization Task , 2007, CVHI.

[5]  Peter B. L. Meijer,et al.  An experimental system for auditory image representations , 1992, IEEE Transactions on Biomedical Engineering.

[6]  Paul Bach-y-Rita.  Sensory substitution and qualia , 2002 .

[7]  Malika Auvray,et al.  Perception With Compensatory Devices: From Sensory Substitution to Sensorimotor Extension , 2009, Cogn. Sci..

[8]  Paul Bach-y-Rita,et al.  Brain plasticity: ‘visual’ acuity of blind persons via the tongue , 2001, Brain Research.

[9]  M. Wanet-Defalque,et al.  Auditory coding of visual patterns for the blind. , 1999, Perception.

[10]  J. O'Regan,et al.  Learning to Perceive with a Visuo — Auditory Substitution System: Localisation and Object Recognition with ‘The Voice’ , 2007, Perception.

[11]  Jack M. Loomis,et al.  Tactile letter recognition under different modes of stimulus presentation , 1974 .

[12]  J C Craig,et al.  Tactile letter recognition: Pattern duration and modes of pattern generation , 1981, Perception & psychophysics.

[13]  P. Stoerig,et al.  Image-to-sound conversion: experience-induced plasticity in auditory cortex of blindfolded adults , 2005, Experimental Brain Research.

[14]  Guy Azémar.  L’homme asymétrique : Gauchers et droitiers face à face , 2003 .

[15]  Marie-Chantal Wanet-Defalque,et al.  Auditory substitution of vision: pattern recognition by the blind , 2001 .

[16]  C. Trullemans,et al.  A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution , 1998, IEEE Transactions on Biomedical Engineering.

[17]  C. Lenay,et al.  SENSORY SUBSTITUTION: LIMITS AND PERSPECTIVES , 2003 .

[18]  D. J. Povel,et al.  Auditory perception and speech: Volume 1, chapter 7, pp. 377–408 by Ira J. Hirsh , 1990 .

[19]  S. Coren,et al.  Lateral preferences and human behavior , 1981 .

[20]  K. C. Persaud,et al.  Blind subjects construct conscious mental images of visual scenes encoded in musical form , 2000, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[21]  S. S. Stevens,et al.  Handbook of experimental psychology , 1951 .

[22]  K. O’regan,et al.  There is something out there: distal attribution in sensory substitution, twenty years later. , 2005, Journal of integrative neuroscience.

[23]  K. Kaczmarek,et al.  Pattern identification and perceived stimulus quality as a function of stimulation waveform on a fingertip-scanned electrotactile display , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[24]  J. Cronly-Dillon,et al.  The perception of visual images encoded in musical form: a study in cross-modality information transfer , 1999, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[25]  G Jansson,et al.  Tactile guidance of movement. , 1983, The International journal of neuroscience.