Active sensory substitution allows fast learning via effective motor-sensory strategies

Summary We examined the development of new sensing abilities in adults by training participants to perceive remote objects through their fingers. Using an Active-Sensing based sensory Substitution device (ASenSub), participants quickly learned to perceive fast via the new modality and preserved their high performance for more than 20 months. Both sighted and blind participants exhibited almost complete transfer of performance from 2D images to novel 3D physical objects. Perceptual accuracy and speed using the ASenSub were, on average, 300% and 600% better than previous reports for 2D images and 3D objects. This improvement is attributed to the ability of the participants to employ their own motor-sensory strategies. Sighted participants dominant strategy was based on motor-sensory convergence on the most informative regions of objects, similarly to fixation patterns in vision. Congenitally, blind participants did not show such a tendency, and many of their exploratory procedures resembled those observed with natural touch.

[1]  E Abravanel,et al.  Intersensory Integration of Spatial Position during Early Childhood , 1968, Perceptual and motor skills.

[2]  J. Norman,et al.  Blindness enhances tactile acuity and haptic 3-D shape discrimination , 2011, Attention, perception & psychophysics.

[3]  Jamie Ward,et al.  Seeing with Sound? Exploring Different Characteristics of a Visual-to-Auditory Sensory Substitution Device , 2011, Perception.

[4]  G. Hauske,et al.  Object and scene analysis by saccadic eye-movements: an investigation with higher-order statistics. , 2000, Spatial vision.

[5]  Daniel Smilek,et al.  Geometric Shape Detection with Soundview , 2004, ICAD.

[6]  Arthur C. Grant,et al.  Tactile perception in blind Braille readers: A psychophysical study of acuity and hyperacuity using gratings and dot patterns , 2000, Perception & psychophysics.

[7]  Shachar Maidenbaum,et al.  Author's Personal Copy Neuroscience and Biobehavioral Reviews Sensory Substitution: Closing the Gap between Basic Research and Widespread Practical Visual Rehabilitation Author's Personal Copy , 2022 .

[8]  M. Heller Picture and Pattern Perception in the Sighted and the Blind: The Advantage of the Late Blind , 1989, Perception.

[9]  P Bach-y-Rita,et al.  Transmission of pictorial information through the skin. , 1973, Advances in biological and medical physics.

[10]  A. Amedi,et al.  The large-scale organization of "visual" streams emerges without visual experience. , 2012, Cerebral cortex.

[11]  M. Ptito,et al.  Cross-modal plasticity revealed by electrotactile stimulation of the tongue in the congenitally blind. , 2005, Brain : a journal of neurology.

[12]  Marie-Chantal Wanet-Defalque,et al.  Auditory Coding of Visual Patterns for the Blind , 1999 .

[13]  R. Klatzky,et al.  Haptic classification of common objects: Knowledge-driven exploration , 1990, Cognitive Psychology.

[14]  M. Mead,et al.  Cybernetics , 1953, The Yale Journal of Biology and Medicine.

[15]  E. Ahissar,et al.  Perception as a closed-loop convergence process , 2016, eLife.

[16]  Hannes P. Saal,et al.  Spatial and temporal codes mediate the tactile perception of natural textures , 2014, 2014 IEEE Haptics Symposium (HAPTICS).

[17]  D. Beebe,et al.  A microfabricated electrostatic haptic display for persons with visual impairments. , 1998, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[18]  D. Goldreich,et al.  Tactile Spatial Acuity Enhancement in Blindness: Evidence for Experience-Dependent Mechanisms , 2011, The Journal of Neuroscience.

[19]  G Guarniero,et al.  Experience of Tactile Vision , 1974, Perception.

[20]  P W Davidson,et al.  Haptic judgments of curvature by blind and sighted humans. , 1972, Journal of experimental psychology.

[21]  Ehud Ahissar,et al.  Importance of Temporal Cues for Tactile Spatial- Frequency Discrimination , 2001, The Journal of Neuroscience.

[22]  J M Kennedy,et al.  Children's tactual exploration and copying without vision , 2001, International journal of rehabilitation research. Internationale Zeitschrift fur Rehabilitationsforschung. Revue internationale de recherches de readaptation.

[23]  Johannes Schemmel,et al.  The virtual haptic display: A device for exploring 2-D virtual shapes in the tactile modality , 2007, Behavior research methods.

[24]  James C Craig,et al.  Perceptual Constancy of Texture Roughness in the Tactile System , 2011, The Journal of Neuroscience.

[25]  E Ahissar,et al.  Oscillatory activity of single units in a somatosensory cortex of an awake monkey and their possible role in texture analysis. , 1990, Proceedings of the National Academy of Sciences of the United States of America.

[26]  Mark Hollins,et al.  The coding of roughness. , 2007, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[27]  J. O'Regan,et al.  Learning to Perceive with a Visuo — Auditory Substitution System: Localisation and Object Recognition with ‘The Voice’ , 2007, Perception.

[28]  Peter König,et al.  Salient features in gaze-aligned recordings of human visual input during free exploration of natural environments. , 2008, Journal of vision.

[29]  J. C. Bliss,et al.  Sensory aids for the blind: A challenging problem with lessons for the future , 1970 .

[30]  Hannes P. Saal,et al.  Millisecond Precision Spike Timing Shapes Tactile Perception , 2012, The Journal of Neuroscience.

[31]  Yon Visell,et al.  Tactile sensory substitution: Models for enaction in HCI , 2009, Interact. Comput..

[32]  K. Rovira,et al.  Mental rotation in blind and sighted adolescents: The effects of haptic strategies , 2011 .

[33]  K. Sathian,et al.  Tactile sensing of surface features , 1989, Trends in Neurosciences.

[34]  Ali Borji,et al.  What stands out in a scene? A study of human explicit saliency judgment , 2013, Vision Research.

[35]  K. Sathian,et al.  Temporal Cues Contribute to Tactile Perception of Roughness , 2001, The Journal of Neuroscience.

[36]  E. Ahissar,et al.  Motor-Sensory Confluence in Tactile Perception , 2012, The Journal of Neuroscience.

[37]  M. Land,et al.  The Roles of Vision and Eye Movements in the Control of Activities of Daily Living , 1998, Perception.

[38]  W T Powers,et al.  Feedback: Beyond Behaviorism , 1973, Science.

[39]  Katia Rovira,et al.  Learning to recognize shapes with a sensory substitution system: A longitudinal study with 4 non-sighted adolescents , 2010, 2010 IEEE 9th International Conference on Development and Learning.

[40]  Valerie Brown,et al.  Eye scanning of multi-element displays: II. Saccade planning , 2006, Vision Research.

[41]  M. Rucci,et al.  Microsaccades Precisely Relocate Gaze in a High Visual Acuity Task , 2010, Nature Neuroscience.

[42]  Ehud Ahissar,et al.  Figuring Space by Time , 2001, Neuron.

[43]  Samuel Lebaz,et al.  Haptic identification of raised-line drawings: high visuospatial imagers outperform low visuospatial imagers , 2012, Psychological research.

[44]  Delphine Picard,et al.  Haptic identification of raised-line drawings when categorical information is given: A comparison between visually impaired and sighted children , 2014 .

[45]  L. Itti,et al.  Search Goal Tunes Visual Features Optimally , 2007, Neuron.

[46]  A. Noë,et al.  A sensorimotor account of vision and visual consciousness. , 2001, The Behavioral and brain sciences.

[47]  Harry Knoors,et al.  Haptic object matching by blind and sighted adults and children. , 2012, Acta psychologica.

[48]  J. Craig,et al.  Shape Invariant Coding of Motion Direction in Somatosensory Cortex , 2010, PLoS biology.

[49]  A. Gale,et al.  Eye Movement Strategies Involved in Face Perception , 1977, Perception.

[50]  Ehud Ahissar,et al.  Tactile Substitution for Vision , 2015, Scholarpedia.

[51]  Derrick J. Parkhurst,et al.  Scene content selected by active vision. , 2003, Spatial vision.

[52]  L. Stark,et al.  Scanpaths in Eye Movements during Pattern Perception , 1971, Science.

[53]  C. Kennard,et al.  The role of visual salience in directing eye movements in visual object agnosia , 2009, Current Biology.

[54]  Á. Pascual-Leone,et al.  The metamodal organization of the brain. , 2001, Progress in brain research.

[55]  P Reinagel,et al.  Natural scene statistics at the centre of gaze. , 1999, Network.

[56]  Sandrine Russier,et al.  Haptic Discrimination of Two-Dimensional Raised-Line Shapes by Blind and Sighted Adults , 1999 .

[57]  C. Lenay,et al.  SENSORY SUBSTITUTION: LIMITS AND PERSPECTIVES , 2003 .

[58]  Gavan Lintern,et al.  Dynamic patterns: The self-organization of brain and behavior , 1997, Complex.

[59]  Ehud Ahissar,et al.  Seeing via Miniature Eye Movements: A Dynamic Hypothesis for Vision , 2012, Front. Comput. Neurosci..

[60]  BENJAMIN WHITE,et al.  Vision Substitution by Tactile Image Projection , 1969, Nature.

[61]  Scott Cheng-Hsin Yang,et al.  Active sensing in the categorization of visual patterns , 2016, eLife.

[62]  Ingrid M. Kanics,et al.  Tactile Acuity is Enhanced in Blindness , 2003, The Journal of Neuroscience.

[63]  K. Johnson,et al.  Neural coding of tactile texture: comparison of spatial and temporal mechanisms for roughness perception , 1992, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[64]  P. Bach-y-Rita,et al.  Form perception with a 49-point electrotactile stimulus array on the tongue: a technical note. , 1998, Journal of rehabilitation research and development.

[65]  M. Wanet-Defalque,et al.  Auditory coding of visual patterns for the blind. , 1999, Perception.

[66]  P W Davidson,et al.  Haptic equivalence matching of curvature by blind and sighted humans. , 1974, Journal of experimental psychology.

[67]  Ehud Ahissar,et al.  Closed loop motor-sensory dynamics in human vision , 2020, PloS one.

[68]  F. Rösler,et al.  Event-related potentials during auditory and somatosensory discrimination in sighted and blind human subjects. , 1996, Brain research. Cognitive brain research.

[69]  Charles Lenay,et al.  A comparison of two methods of scaling on form perception via a haptic interface , 2005, ICMI '05.

[70]  Joshua Atkins,et al.  Perceiving graphical and pictorial information via touch and hearing , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[71]  K G O'Bryan,et al.  Eye movements, perceptual activity, and conservation development. , 1971, Journal of experimental child psychology.

[72]  William M. Stern,et al.  Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex , 2007, Nature Neuroscience.

[73]  Y. Hatwell,et al.  Etude de quelques illusions géométriques tactiles chez les aveugles , 1960 .