EyeMusic: Introducing a "visual" colorful experience for the blind using auditory sensory substitution.

PURPOSE Sensory-substitution devices (SSDs) provide auditory or tactile representations of visual information. These devices often generate unpleasant sensations and mostly lack color information. We present here a novel SSD aimed at addressing these issues. METHODS We developed the EyeMusic, a novel visual-to-auditory SSD for the blind, providing both shape and color information. Our design uses musical notes on a pentatonic scale generated by natural instruments to convey the visual information in a pleasant manner. A short behavioral protocol was utilized to train the blind to extract shape and color information, and test their acquired abilities. Finally, we conducted a survey and a comparison task to assess the pleasantness of the generated auditory stimuli. RESULTS We show that basic shape and color information can be decoded from the generated auditory stimuli. High performance levels were achieved by all participants following as little as 2-3 hours of training. Furthermore, we show that users indeed found the stimuli pleasant and potentially tolerable for prolonged use. CONCLUSIONS The novel EyeMusic algorithm provides an intuitive and relatively pleasant way for the blind to extract shape and color information. We suggest that this might help facilitating visual rehabilitation because of the added functionality and enhanced pleasantness.

[1]  Jamie Ward,et al.  Seeing with Sound? Exploring Different Characteristics of a Visual-to-Auditory Sensory Substitution Device , 2011, Perception.

[2]  Thomas D. Wright,et al.  The evolution of a visual-to-auditory sensory substitution device using interactive genetic algorithms , 2013, Quarterly journal of experimental psychology.

[3]  Ron Kupers,et al.  Navigation with a sensory substitution device in congenitally blind individuals , 2011, Neuroreport.

[4]  K A Kaczmarek,et al.  The tongue display unit (TDU) for electrotactile spatiotemporal pattern presentation , 2011, Sci. Iran..

[5]  Antonio Torralba,et al.  Recognizing indoor scenes , 2009, CVPR.

[6]  S Levy-Tzedek,et al.  Fast, accurate reaching movements with a visual-to-auditory sensory substitution device. , 2012, Restorative neurology and neuroscience.

[7]  Kristoffer Hougaard Madsen,et al.  Neural correlates of virtual route recognition in congenital blindness , 2010, Proceedings of the National Academy of Sciences.

[8]  Amir Amedi,et al.  Reading with Sounds: Sensory Substitution Selectively Activates the Visual Word Form Area in the Blind , 2012, Neuron.

[9]  B. Sabel,et al.  Restoration of vision II: residual functions and training-induced visual field enlargement in brain-damaged patients. , 1999, Restorative neurology and neuroscience.

[10]  J. O'Regan,et al.  Learning to Perceive with a Visuo — Auditory Substitution System: Localisation and Object Recognition with ‘The Voice’ , 2007, Perception.

[11]  Shachar Maidenbaum,et al.  Cross-sensory transfer of sensory-motor information: visuomotor learning affects performance on an audiomotor task, using sensory-substitution , 2012, Scientific Reports.

[12]  K. Sathian,et al.  Cross-modal plasticity of tactile perception in blindness. , 2010, Restorative neurology and neuroscience.

[13]  P. Sinha,et al.  Contribution of Color to Face Recognition , 2002, Perception.

[14]  A. Oliva,et al.  Diagnostic colours contribute to the early stages of scene categorization: Behavioural and neurophysiological evidence , 2005 .

[15]  Keh-chung Lin,et al.  Pleasant music improves visual attention in patients with unilateral neglect after stroke , 2013, Brain injury.

[16]  José Manuel Ferrández,et al.  Bioinspired Applications in Artificial and Natural Computation, Third International Work-Conference on the Interplay Between Natural and Artificial Computation, IWINAC 2009, Santiago de Compostela, Spain, June 22-26, 2009, Proceedings, Part II , 2009, IWINAC.

[17]  Amir Amedi,et al.  ‘Visual’ Acuity of the Congenitally Blind Using Visual-to-Auditory Sensory Substitution , 2012, PloS one.

[18]  Shachar Maidenbaum,et al.  Plasticity in Sensory Systems: Applying Plasticity to Visual Rehabilitation in Adulthood , 2012 .

[19]  Kees van den Doel,et al.  SOUNDVIEW: SENSING COLOR IMAGES BY KINESTHETIC AUDIO , 2003 .

[20]  P. Stoerig,et al.  Seeing ‘Where’ through the Ears: Effects of Learning-by-Doing and Long-Term Sensory Deprivation on Localization Based on Image-to-Sound Substitution , 2008, PloS one.

[21]  Pia Rotshtein,et al.  Pleasant music overcomes the loss of awareness in patients with visual neglect , 2009, Proceedings of the National Academy of Sciences.

[22]  T. Särkämö,et al.  Music listening after stroke: beneficial effects and potential neural mechanisms , 2012, Annals of the New York Academy of Sciences.

[23]  Peter B. L. Meijer,et al.  An experimental system for auditory image representations , 1992, IEEE Transactions on Biomedical Engineering.

[24]  Peter B. L. Meijer,et al.  Visual experiences in the blind induced by an auditory sensory substitution device , 2010, Consciousness and Cognition.

[25]  C. Trullemans,et al.  A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution , 1998, IEEE Transactions on Biomedical Engineering.

[26]  T. Griffiths,et al.  Mapping unpleasantness of sounds to their auditory representation. , 2008, The Journal of the Acoustical Society of America.

[27]  A. Volder,et al.  Pattern recognition using a device substituting audition for vision in blindfolded sighted subjects , 2007, Neuropsychologia.

[28]  Guido Bologna,et al.  Transforming 3D Coloured Pixels into Musical Instrument Notes for Vision Substitution Applications , 2007, EURASIP J. Image Video Process..

[29]  K. Petersson,et al.  The role of color information on object recognition: a review and meta-analysis. , 2011, Acta psychologica.

[30]  Sethuraman Panchanathan,et al.  Learning and perceiving colors haptically , 2006, Assets '06.

[31]  G. Dagnelie.  Retinal implants: emergence of a multidisciplinary field. , 2012, Current opinion in neurology.

[32]  Stephen Gould,et al.  Decomposing a scene into geometric and semantically consistent regions , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[33]  Malika Auvray,et al.  Perception With Compensatory Devices: From Sensory Substitution to Sensorimotor Extension , 2009, Cogn. Sci..

[34]  A. Amedi,et al.  The brain as a flexible task machine: implications for visual rehabilitation using noninvasive vs. invasive approaches. , 2012, Current opinion in neurology.

[35]  Keh-chung Lin,et al.  Listening to classical music ameliorates unilateral neglect after stroke. , 2013, The American journal of occupational therapy : official publication of the American Occupational Therapy Association.

[36]  A. D. De Volder,et al.  Vision substitution and depth perception: Early blind subjects experience visual perspective through their ears , 2010, Disability and rehabilitation. Assistive technology.

[37]  BENJAMIN WHITE,et al.  Vision Substitution by Tactile Image Projection , 1969, Nature.

[38]  Ingrid M. Kanics,et al.  Tactile Acuity is Enhanced in Blindness , 2003, The Journal of Neuroscience.