VES: A Mixed-Reality System to Assist Multisensory Spatial Perception and Cognition for Blind and Visually Impaired People

In this paper, the Virtually Enhanced Senses (VES) System is described. It is an ARCore-based, mixed-reality system meant to assist blind and visually impaired people’s navigation. VES operates in indoor and outdoor environments without any previous in-situ installation. It provides users with specific, runtime-configurable stimuli according to their pose, i.e., position and orientation, and the information of the environment recorded in a virtual replica. It implements three output data modalities: wall-tracking assistance, acoustic compass, and a novel sensory substitution algorithm, Geometry-based Virtual Acoustic Space (GbVAS). The multimodal output of this algorithm takes advantage of natural human perception encoding of spatial data. Preliminary experiments of GbVAS have been conducted with sixteen subjects in three different scenarios, demonstrating basic orientation and mobility skills after six minutes training.

[1]  José L. González-Mora,et al.  Development of a New Space Perception System for Blind People, Based on the Creation of a Virtual Acoustic Space , 1999, IWANN.

[2]  R. Klatzky,et al.  - Sensory Substitution of Vision: Importance of Perceptual and Cognitive Processing , 2018, Assistive Technology for Blindness and Low Vision.

[3]  J.L. Gonzalez-Mora,et al.  Seeing the world by hearing: Virtual Acoustic Space (VAS) a new space perception system for blind people. , 2006, 2006 2nd International Conference on Information & Communication Technologies.

[4]  Carlos Martín,et al.  Differences between blind people's cognitive maps after proximity and distant exploration of virtual environments , 2017, Comput. Hum. Behav..

[5]  Paul Bach-y-Rita,et al.  Brain plasticity: ‘visual’ acuity of blind persons via the tongue , 2001, Brain Research.

[6]  Masatoshi Ishikawa,et al.  Augmenting spatial awareness with Haptic Radar , 2006, 2006 10th IEEE International Symposium on Wearable Computers.

[7]  Simon Meers,et al.  A vision system for providing 3D perception of the environment via transcutaneous electro-neural stimulation , 2004, Proceedings. Eighth International Conference on Information Visualisation, 2004. IV 2004..

[8]  Willis J. Tompkins,et al.  A Tactile Vision-Substitution System for the Blind: Computer-Controlled Partial Image Sequencing , 1985, IEEE Transactions on Biomedical Engineering.

[9]  C. Thinus-Blanc,et al.  Representation of space in blind persons: vision as a spatial sense? , 1997, Psychological bulletin.

[10]  Tyler Thrash,et al.  Spatial navigation by congenitally blind individuals , 2015, Wiley interdisciplinary reviews. Cognitive science.

[11]  Shachar Maidenbaum,et al.  Author's Personal Copy Neuroscience and Biobehavioral Reviews Sensory Substitution: Closing the Gap between Basic Research and Widespread Practical Visual Rehabilitation Author's Personal Copy , 2022 .

[12]  Daniela Massiceti,et al.  Stereosonic vision: Exploring visual-to-auditory sensory substitution mappings in an immersive virtual reality navigation paradigm , 2018, PloS one.

[13]  Rabia Jafri,et al.  Visual and Infrared Sensor Data-Based Obstacle Detection for the Visually Impaired Using the Google Project Tango Tablet Development Kit and the Unity Engine , 2018, IEEE Access.

[14]  Roberta L. Klatzky,et al.  Assisting Wayfinding in Visually Impaired Travelers , 2006 .

[15]  Andreas Hub.  Precise Indoor and Outdoor Navigation for the Blind and Visually Impaired Using Augmented Maps and the TANIA System , 2008 .

[16]  Luca Viganò,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2015, IWSEC 2015.

[17]  Victor R. Schinazi.  Representing space : the development, content and accuracy of mental representations by the blind and visually impaired , 2008 .

[18]  N. Giudice.  Navigating without vision: principles of blind spatial cognition , 2018 .

[19]  Peter B. L. Meijer,et al.  Visual experiences in the blind induced by an auditory sensory substitution device , 2010, Consciousness and Cognition.

[20]  Hironobu Takagi,et al.  NavCog: a navigational cognitive assistant for the blind , 2016, MobileHCI.

[21]  Donald C. Fletcher,et al.  The Functional Performance of the BrainPort V100 Device in Persons who Are Profoundly Blind , 2016 .

[22]  Peter B. L. Meijer,et al.  An experimental system for auditory image representations , 1992, IEEE Transactions on Biomedical Engineering.

[23]  Masatoshi Ishikawa,et al.  Virtual Haptic Radar , 2009, SIGGRAPH ASIA '09.

[24]  Lindsay Evett,et al.  Virtual environment navigation with look-around mode to explore new real spaces by people who are blind , 2018, Disability and rehabilitation.

[25]  Aaron M. Dollar,et al.  Design and Evaluation of Shape-Changing Haptic Interfaces for Pedestrian Navigation Assistance , 2017, IEEE Transactions on Haptics.

[26]  Alvaro Araujo,et al.  Navigation Systems for the Blind and Visually Impaired: Past Work, Challenges, and Open Problems , 2019, Sensors.

[27]  Yi Zhu,et al.  An ARCore Based User Centric Assistive Navigation System for Visually Impaired People , 2019, Applied Sciences.

[28]  Michael J. Proulx,et al.  How well do you see what you hear? The acuity of visual-to-auditory sensory substitution , 2013, Front. Psychol..