The Topo-Speech sensory substitution system as a method of conveying spatial information to the blind and vision impaired

Humans, like most animals, integrate sensory input in the brain from different sensory modalities. Yet humans are distinct in their ability to grasp symbolic input, which is interpreted into a cognitive mental representation of the world. This representation merges with external sensory input, providing modality integration of a different sort. This study evaluates the Topo-Speech algorithm in the blind and visually impaired. The system provides spatial information about the external world by applying sensory substitution alongside symbolic representations in a manner that corresponds with the unique way our brains acquire and process information. This is done by conveying spatial information, customarily acquired through vision, through the auditory channel, in a combination of sensory (auditory) features and symbolic language (named/spoken) features. The Topo-Speech sweeps the visual scene or image and represents objects’ identity by employing naming in a spoken word and simultaneously conveying the objects’ location by mapping the x-axis of the visual scene or image to the time it is announced and the y-axis by mapping the location to the pitch of the voice. This proof of concept study primarily explores the practical applicability of this approach in 22 visually impaired and blind individuals. The findings showed that individuals from both populations could effectively interpret and use the algorithm after a single training session. The blind showed an accuracy of 74.45%, while the visually impaired had an average accuracy of 72.74%. These results are comparable to those of the sighted, as shown in previous research, with all participants above chance level. As such, we demonstrate practically how aspects of spatial information can be transmitted through non-visual channels. To complement the findings, we weigh in on debates concerning models of spatial knowledge (the persistent, cumulative, or convergent models) and the capacity for spatial representation in the blind. We suggest the present study’s findings support the convergence model and the scenario that posits the blind are capable of some aspects of spatial representation as depicted by the algorithm comparable to those of the sighted. Finally, we present possible future developments, implementations, and use cases for the system as an aid for the blind and visually impaired.

[1]  M. Ptito,et al.  Neural substrates of spatial processing and navigation in blindness: An activation likelihood estimation meta-analysis , 2022, Frontiers in Neuroscience.

[2]  Maria Bianca Amadeo,et al.  Effects of audio-motor training on spatial representations in long-term late blindness , 2022, Neuropsychologia.

[3]  S. Lomber,et al.  Do blind people hear better? , 2022, Trends in Cognitive Sciences.

[4]  A. Amedi,et al.  A case study in phenomenology of visual experience with retinal prosthesis versus visual-to-auditory sensory substitution , 2022, Neuropsychologia.

[5]  A. Amedi,et al.  Effects of training and using an audio-tactile sensory substitution device on speech-in-noise understanding , 2022, Scientific Reports.

[6]  Blake S. Wilson,et al.  Harnessing the power of artificial intelligence to transform hearing healthcare and research , 2021, Nature Machine Intelligence.

[7]  J. Kim,et al.  Shared understanding of color among sighted and blind adults , 2021, Proceedings of the National Academy of Sciences.

[8]  A. Amedi,et al.  Backward spatial perception can be augmented through a novel visual-to-auditory sensory substitution algorithm , 2021, Scientific Reports.

[9]  M. Ptito,et al.  Brain-Machine Interfaces to Assist the Blind , 2021, Frontiers in Human Neuroscience.

[10]  Michał Wierzchoń,et al.  Visual Echolocation Concept for the Colorophone Sensory Substitution Device Using Virtual Reality , 2021, Sensors.

[11]  E. Newport,et al.  Spoken language comprehension activates the primary visual cortex , 2020, bioRxiv.

[12]  L. Merabet,et al.  The Effect of Blindness on Spatial Asymmetries , 2020, Brain sciences.

[13]  Valeria Occelli,et al.  General Enhancement of Spatial Hearing in Congenitally Blind People , 2020, Psychological science.

[14]  M. Gori,et al.  The Impact of Vision Loss on Allocentric Spatial Coding , 2020, Frontiers in Neuroscience.

[15]  Monica Gori,et al.  Spatial metric in blindness: behavioural and cortical processing , 2019, Neuroscience & Biobehavioral Reviews.

[16]  M. Gori,et al.  The Role of Vision on Spatial Competence , 2019, Visual Impairment and Blindness - What We Know and What We Have to Know.

[17]  M. Gori,et al.  Audio-Motor Training Enhances Auditory and Proprioceptive Functions in the Blind Adult , 2019, Front. Neurosci..

[18]  Fabian Hutmacher Why Is There So Much More Research on Vision Than on Any Other Sensory Modality? , 2019, Front. Psychol..

[19]  P. Bruns The Ventriloquist Illusion as a Tool to Study Multisensory Processing: An Update , 2019, Front. Integr. Neurosci..

[20]  Amir Amedi,et al.  The Topo-Speech Algorithm: An intuitive Sensory Substitution for Spatial Information , 2019, 2019 International Conference on Virtual Rehabilitation (ICVR).

[21]  Amir Amedi,et al.  Immediate improvement of speech-in-noise perception through multisensory stimulation via an auditory to tactile sensory substitution , 2019, Restorative neurology and neuroscience.

[22]  G. Giammari,et al.  Audio motor training improves mobility and spatial cognition in visually impaired children , 2019, Scientific Reports.

[23]  M. Gori,et al.  Shape Recognition With Sounds: Improvement in Sighted Individuals After Audio-Motor Training. , 2019, Multisensory research.

[24]  Matthew G. Crowson,et al.  Machine Learning and Cochlear Implantation-A Structured Review of Opportunities and Challenges. , 2019, Otology & neurotology : official publication of the American Otological Society, American Neurotology Society [and] European Academy of Otology and Neurotology.

[25]  M. Gori,et al.  Audio Feedback Associated With Body Movement Enhances Audio and Somatosensory Spatial Representation , 2018, Front. Integr. Neurosci..

[26]  Monica Gori,et al.  Audio Motor Training at the Foot Level Improves Space Representation , 2017, Front. Integr. Neurosci..

[27]  G. Baud-Bovy,et al.  Multisensory Rehabilitation Training Improves Spatial Perception in Totally but Not Partially Visually Deprived Children , 2017, Front. Integr. Neurosci..

[28]  M. Bedny Evidence from Blindness for a Cognitively Pluripotent Cortex , 2017, Trends in Cognitive Sciences.

[29]  M. Gori,et al.  The Impact of Early Visual Deprivation on Spatial Hearing: A Comparison between Totally and Partially Visually Deprived Children , 2017, Front. Psychol..

[30]  L. Merabet,et al.  Multimodal MR-imaging reveals large-scale structural and functional connectivity changes in profound early blindness , 2017, PloS one.

[31]  M. Gori,et al.  Auditory Spatial Recalibration in Congenital Blind Individuals , 2017, Front. Neurosci..

[32]  Shachar Maidenbaum,et al.  Waist-up protection for blind individuals using the EyeCane as a primary and secondary mobility aid , 2017, Restorative neurology and neuroscience.

[33]  Lore Thaler,et al.  Echolocation in humans: an overview. , 2016, Wiley interdisciplinary reviews. Cognitive science.

[34]  A. Postma,et al.  Multisensory Perception and the Coding of Space , 2016 .

[35]  Ómar I. Jóhannesson,et al.  Designing sensory-substitution devices: Principles, pitfalls and potential , 2016, Restorative neurology and neuroscience.

[36]  Tyler Thrash,et al.  Spatial navigation by congenitally blind individuals , 2015, Wiley interdisciplinary reviews. Cognitive science.

[37]  Shachar Maidenbaum,et al.  Integration and binding in rehabilitative sensory substitution: Increasing resolution using a new Zooming-in approach , 2015, Restorative neurology and neuroscience.

[38]  Shachar Maidenbaum,et al.  Navigation Using Sensory Substitution in Real and Virtual Mazes , 2015, PloS one.

[39]  Shachar Maidenbaum,et al.  A number-form area in the blind , 2015, Nature Communications.

[40]  P. Pietrini,et al.  Mind the blind brain to understand the sighted one! Is there a supramodal cortical functional architecture? , 2014, Neuroscience & Biobehavioral Reviews.

[41]  M. Ptito,et al.  Compensatory plasticity and cross-modal reorganization following early visual deprivation , 2014, Neuroscience & Biobehavioral Reviews.

[42]  Shachar Maidenbaum,et al.  Author's Personal Copy Neuroscience and Biobehavioral Reviews Sensory Substitution: Closing the Gap between Basic Research and Widespread Practical Visual Rehabilitation Author's Personal Copy , 2022 .

[43]  G. Sandini,et al.  Impairment of auditory spatial localization in congenitally blind human subjects , 2013, Brain : a journal of neurology.

[44]  Shachar Maidenbaum,et al.  EyeMusic: Introducing a "visual" colorful experience for the blind using auditory sensory substitution. , 2014, Restorative neurology and neuroscience.

[45]  Shachar Maidenbaum,et al.  The "EyeCane", a new electronic travel aid for the blind: Technology, behavior & swift learning. , 2014, Restorative neurology and neuroscience.

[46]  Pavel Benes,et al.  Visual field. , 2013, Collegium antropologicum.

[47]  G. Medioni,et al.  Smart image processing system for retinal prosthesis , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[48]  Amir Amedi,et al.  Reading with Sounds: Sensory Substitution Selectively Activates the Visual Word Form Area in the Blind , 2012, Neuron.

[49]  Nick Barnes,et al.  The role of computer vision in prosthetic vision , 2012, Image Vis. Comput..

[50]  C. Spence,et al.  The Colavita Visual Dominance Effect , 2012 .

[51]  Marijn E. Struiksma,et al.  Spatial Language Processing in the Blind: Evidence for a Supramodal Representation and Cortical Reorganization , 2011, PloS one.

[52]  Mats E Nilsson,et al.  Human Echolocation: Pitch versus Loudness Information , 2011, Perception.

[53]  Heiko Hecht,et al.  Fashion Versus Perception: The Impact of Surface Lightness on the Perceived Dimensions of Interior Space , 2011, Hum. Factors.

[54]  Ron Kupers,et al.  Navigation with a sensory substitution device in congenitally blind individuals , 2011, Neuroreport.

[55]  Kemal Yildirim,et al.  Effects of Interior Colors on Mood and Preference: Comparisons of Two Living Rooms , 2011, Perceptual and motor skills.

[56]  T. Vecchi,et al.  Blind Vision: The Neuroscience of Visual Impairment , 2011 .

[57]  M. Denis,et al.  Structural properties of spatial representations in blind people: Scanning images constructed from haptic exploration or from locomotion in a 3-D audio virtual environment , 2010, Memory & cognition.

[58]  Peter B. L. Meijer,et al.  Visual experiences in the blind induced by an auditory sensory substitution device , 2010, Consciousness and Cognition.

[59]  Marijn E. Struiksma,et al.  What is the link between language and spatial images? Behavioral and neural findings in blind and sighted individuals. , 2009, Acta psychologica.

[60]  M. Lassonde,et al.  Cross-modal plasticity for the spatial processing of sounds in visually deprived subjects , 2008, Experimental Brain Research.

[61]  William M. Stern,et al.  Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex , 2007, Nature Neuroscience.

[62]  Shai Avidan,et al.  Blind Vision , 2006, ECCV.

[63]  Olivier Collignon,et al.  Improved selective and divided spatial attention in early blind subjects , 2006, Brain Research.

[64]  Peter B. L. Meijer,et al.  Neural correlates of visual-to-auditory sensory substitution in proficient blind users. , 2005 .

[65]  R. Zatorre,et al.  A Functional Neuroimaging Study of Sound Localization: Visual Cortex Activity Predicts Performance in Early-Blind Individuals , 2005, PLoS biology.

[66]  L. Merabet,et al.  What blindness can tell us about seeing again: merging neuroplasticity and neuroprostheses , 2005, Nature Reviews Neuroscience.

[67]  Franco Lepore,et al.  Early- and Late-Onset Blind Individuals Show Supra-Normal Auditory Abilities in Far-Space , 2004, Current Biology.

[68]  E. Zohary,et al.  Transcranial magnetic stimulation of the occipital pole interferes with verbal processing in blind subjects , 2004, Nature Neuroscience.

[69]  R. Zatorre,et al.  Neuropsychology: Pitch discrimination in the early blind , 2004, Nature.

[70]  C. Spence,et al.  Early Vision Impairs Tactile Perception in the Blind , 2004, Current Biology.

[71]  M. Lassonde,et al.  Blind subjects process auditory spectral cues more efficiently than sighted individuals , 2004, Experimental Brain Research.

[72]  David S. Vogel,et al.  Cerebral lateralization of spatial abilities: A meta-analysis , 2003, Brain and Cognition.

[73]  R. Malach,et al.  Early ‘visual’ cortex activation correlates with superior verbal memory performance in the blind , 2003, Nature Neuroscience.

[74]  Roberta L Klatzky,et al.  Spatial updating of locations specified by 3-d sound and spatial language. , 2002, Journal of experimental psychology. Learning, memory, and cognition.

[75]  T. Hendler,et al.  Visuo-haptic object-related activation in the ventral visual pathway , 2001, Nature Neuroscience.

[76]  Á. Pascual-Leone,et al.  Tactile spatial resolution in blind Braille readers , 2000, Neurology.

[77]  S. Hillyard,et al.  Improved auditory spatial tuning in blind humans , 1999, Nature.

[78]  M. Paré,et al.  Early-blind human subjects localize sound sources better than sighted subjects , 1998, Nature.

[79]  M. Hallett,et al.  Neural networks for Braille reading by the blind. , 1998 .

[80]  Alvaro Pascual-Leone,et al.  Cortical plasticity associated with Braille learning , 1998, Trends in Cognitive Sciences.

[81]  Karl J. Friston,et al.  Different activation patterns in the visual cortex of late and congenitally blind subjects. , 1998, Brain : a journal of neurology.

[82]  Florence Gaunet,et al.  Early-Blind Subjects' Spatial Representation of Manipulatory Space: Exploratory Strategies and Reaction to Change , 1997, Perception.

[83]  M. Hallett,et al.  Activation of the primary visual cortex by Braille reading in blind subjects , 1996, Nature.

[84]  G K Humphrey,et al.  Spatial knowledge in blind and sighted children. , 1995, Journal of experimental child psychology.

[85]  M. Hallett,et al.  Modulation of motor cortical outputs to the reading hand of braille readers , 1993, Annals of neurology.

[86]  Á. Pascual-Leone,et al.  Plasticity of the sensorimotor cortex representation of the reading finger in Braille readers. , 1993, Brain : a journal of neurology.

[87]  R N Haber,et al.  Properties of spatial representations: Data from sighted and blind subjects , 1993, Perception & psychophysics.

[88]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[89]  M. Posner,et al.  Visual dominance: an information-processing account of its origins and significance. , 1976, Psychological review.

[90]  F. Colavita Human sensory dominance , 1974 .

[91]  Robert A. Wilkin,et al.  National Federation of the Blind , 1958 .