"I Always Wanted to See the Night Sky": Blind User Preferences for Sensory Substitution Devices

Sensory Substitution Devices (SSDs) convert visual information into another sensory channel (e.g. sound) to improve the everyday functioning of blind and visually impaired persons (BVIP). However, the range of possible functions and options for translating vision into sound is largely open-ended. To provide constraints on the design of this technology, we interviewed ten BVIPs who were briefly trained in the use of three novel devices that, collectively, showcase a large range of design permutations. The SSDs include the 'Depth-vOICe,' 'Synaestheatre' and 'Creole' that offer high spatial, temporal, and colour resolutions respectively via a variety of sound outputs (electronic tones, instruments, vocals). The participants identified a range of practical concerns in relation to the devices (e.g. curb detection, recognition, mental effort) but also highlighted experiential aspects. This included both curiosity about the visual world (e.g. understanding shades of colour, the shape of cars, seeing the night sky) and the desire for the substituting sound to be responsive to movement of the device and aesthetically engaging.

[1]  Guido Bologna,et al.  Transforming 3D Coloured Pixels into Musical Instrument Notes for Vision Substitution Applications , 2007, EURASIP J. Image Video Process..

[2]  Ron Kupers,et al.  Navigation with a sensory substitution device in congenitally blind individuals , 2011, Neuroreport.

[3]  Eelke Folmer,et al.  Headlock: a wearable navigation aid that helps blind cane users traverse large open spaces , 2014, ASSETS.

[4]  Robert J. Greenberg,et al.  The Argus® II retinal prosthesis system: An overview , 2013, 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[5]  Andrew J. R. Simpson,et al.  Visual objects in the auditory system in sensory substitution: how much information do we need? , 2014, Multisensory research.

[6]  Peter B. L. Meijer,et al.  Visual experiences in the blind induced by an auditory sensory substitution device , 2010, Consciousness and Cognition.

[7]  Amir Amedi,et al.  ‘Visual’ Acuity of the Congenitally Blind Using Visual-to-Auditory Sensory Substitution , 2012, PloS one.

[8]  Seung-Chan Kim,et al.  Tactile feedback on flat surfaces for the visually impaired , 2012, CHI Extended Abstracts.

[9]  A. Amedi,et al.  Virtually zooming-in with sensory substitution for blind users , 2015, 2015 International Conference on Virtual Rehabilitation (ICVR).

[10]  Peter B. L. Meijer,et al.  An experimental system for auditory image representations , 1992, IEEE Transactions on Biomedical Engineering.

[11]  Shachar Maidenbaum,et al.  EyeMusic: Introducing a "visual" colorful experience for the blind using auditory sensory substitution. , 2014, Restorative neurology and neuroscience.

[12]  Richard E. Ladner,et al.  Design for user empowerment , 2014, CHI Extended Abstracts.

[13]  J. O'Regan,et al.  Learning to Perceive with a Visuo — Auditory Substitution System: Localisation and Object Recognition with ‘The Voice’ , 2007, Perception.

[14]  Uran Oh,et al.  Current and future mobile and wearable device use by people with visual impairments , 2014, CHI.

[15]  P. Stoerig,et al.  Seeing ‘Where’ through the Ears: Effects of Learning-by-Doing and Long-Term Sensory Deprivation on Localization Based on Image-to-Sound Substitution , 2008, PloS one.

[16]  Amy Hurst,et al.  "just let the cane hit it": how the blind and sighted see navigation differently , 2014, ASSETS.

[17]  Jamie Ward,et al.  Seeing with Sound? Exploring Different Characteristics of a Visual-to-Auditory Sensory Substitution Device , 2011, Perception.

[18]  Anne Treisman,et al.  Natural cross-modal mappings between visual and auditory features. , 2011, Journal of vision.

[19]  C. Spence Crossmodal correspondences: A tutorial review , 2011, Attention, perception & psychophysics.

[20]  Roberta L. Klatzky,et al.  Stated Preferences for Components of a Personal Guidance System for Nonvisual Navigation , 2004 .

[21]  Amir Amedi,et al.  Reading with Sounds: Sensory Substitution Selectively Activates the Visual Word Form Area in the Blind , 2012, Neuron.

[22]  L E Marks,et al.  On cross-modal similarity: auditory-visual interactions in speeded discrimination. , 1987, Journal of experimental psychology. Human perception and performance.

[23]  Mohammed Yeasin,et al.  Do you see what I see?: designing a sensory substitution device to access non-verbal modes of communication , 2013, ASSETS.

[24]  Jörg Lewald,et al.  Vertical sound localization in blind humans , 2002, Neuropsychologia.

[25]  Hervé Segond,et al.  Human Spatial Navigation via a Visuo-Tactile Sensory Substitution System , 2005, Perception.

[26]  J.L. Gonzalez-Mora,et al.  Seeing the world by hearing: Virtual Acoustic Space (VAS) a new space perception system for blind people. , 2006, 2006 2nd International Conference on Information & Communication Technologies.

[27]  Jamie Ward,et al.  Cross-Modal Correspondences Enhance Performance on a Colour-to-Sound Sensory Substitution Device. , 2016, Multisensory research.

[28]  Shachar Maidenbaum,et al.  Author's Personal Copy Neuroscience and Biobehavioral Reviews Sensory Substitution: Closing the Gap between Basic Research and Widespread Practical Visual Rehabilitation Author's Personal Copy , 2022 .

[29]  Jamie Ward,et al.  Sensory Substitution Devices as Advanced Sensory Tools , 2018, Sensory Substitution and Augmentation.

[30]  Thomas D. Wright,et al.  Where’s Wally? Audio–visual mismatch directs ocular saccades in sensory substitution , 2012 .

[31]  William M. Stern,et al.  Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex , 2007, Nature Neuroscience.

[32]  Peter B. L. Meijer,et al.  Functional recruitment of visual cortex for sound encoded object identification in the blind , 2009, Neuroreport.

[33]  Michael J. Proulx,et al.  How well do you see what you hear? The acuity of visual-to-auditory sensory substitution , 2013, Front. Psychol..

[34]  Stephen A. Brewster,et al.  Feeling what you hear: tactile feedback for navigation of audio graphs , 2006, CHI.

[35]  S. Hillyard,et al.  Improved auditory spatial tuning in blind humans , 1999, Nature.

[36]  Jung-Kyong Kim,et al.  Generalized Learning of Visual-to-auditory Substitution in Sighted Individuals , 2008 .

[37]  Jamie Ward,et al.  Representing colour through hearing and touch in sensory substitution devices. , 2013, Multisensory research.