Sensorimotor strategies for recognizing geometrical shapes: a comparative study with different sensory substitution devices

The sensorimotor approach proposes that perception is constituted by the mastery of lawful sensorimotor regularities or sensorimotor contingencies (SMCs), which depend on specific bodily characteristics and on actions possibilities that the environment enables and constrains. Sensory substitution devices (SSDs) provide the user information about the world typically corresponding to one sensory modality through the stimulation of another modality. We investigate how perception emerges in novice adult participants equipped with vision-to-auditory SSDs while solving a simple geometrical shape recognition task. In particular, we examine the distinction between apparatus-related SMCs (those originating mostly in properties of the perceptual system) and object-related SMCs (those mostly connected with the perceptual task). We study the sensorimotor strategies employed by participants in three experiments with three different SSDs: a minimalist head-mounted SSD, a traditional, also head-mounted SSD (the vOICe) and an enhanced, hand-held echolocation device. Motor activity and fist-person data are registered and analyzed. Results show that participants are able to quickly learn the necessary skills to distinguish geometric shapes. Comparing the sensorimotor strategies utilized with each SSD we identify differential features of the sensorimotor patterns attributable mostly to the device, which account for the emergence of apparatus-based SMCs. These relate to differences in sweeping strategies between SSDs. We identify, also, components related to the emergence of object-related SMCs. These relate mostly to exploratory movements around the border of a shape. The study provides empirical support for SMC theory and discusses considerations about the nature of perception in sensory substitution.

[1]  A. Noë,et al.  Sensory consciousness explained (better) in terms of ‘corporality’ and ‘alerting capacity’ , 2005 .

[2]  Amir Amedi,et al.  Flexibility and Stability in Sensory Processing Revealed Using Visual-to-Auditory Sensory Substitution , 2014, Cerebral cortex.

[3]  Lore Thaler,et al.  Correlation between vividness of visual imagery and echolocation ability in sighted, echo-naïve people , 2014, Experimental Brain Research.

[4]  Hilla Peretz,et al.  The , 1966 .

[5]  Charles Lenay,et al.  Haptic recognition of shapes at different scales: A comparison of two methods of interaction , 2007, Interact. Comput..

[6]  Peter B. L. Meijer,et al.  Visual experiences in the blind induced by an auditory sensory substitution device , 2010, Consciousness and Cognition.

[7]  V. Dreyer,et al.  Visual acuity. , 1974, Ophthalmologica. Journal international d'ophtalmologie. International journal of ophthalmology. Zeitschrift fur Augenheilkunde.

[8]  Marie-Chantal Wanet-Defalque,et al.  Auditory substitution of vision: pattern recognition by the blind , 2001 .

[9]  Yohan Payan,et al.  Effect of saccades in tongue electrotactile stimulation for vision substitution applications , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[10]  Ned Block,et al.  Tactile sensation via spatial perception , 2003, Trends in Cognitive Sciences.

[11]  William M. Stern,et al.  Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex , 2007, Nature Neuroscience.

[12]  R L Klatzky,et al.  Haptic processing of spatially distributed information , 1988, Perception & psychophysics.

[13]  Melvyn A. Goodale,et al.  The role of head movements in the discrimination of 2-D shape by blind echolocation experts , 2014, Attention, perception & psychophysics.

[14]  Donald M. MacKay,et al.  Visual Stability and Voluntary Eye Movements , 1973 .

[15]  I. G. Bassett,et al.  Echolocation: Measurement of Pitch versus Distance for Sounds Reflected from a Flat Surface , 1964 .

[16]  Sylvia Weir,et al.  Action perception , 1974 .

[17]  Mats E Nilsson,et al.  Human Echolocation: Pitch versus Loudness Information , 2011, Perception.

[18]  Sabine U. König,et al.  The experience of new sensorimotor contingencies by sensory augmentation , 2014, Consciousness and Cognition.

[19]  K. Saberi,et al.  Lateralization thresholds obtained under conditions in which the precedence effect is assumed to operate. , 1990, The Journal of the Acoustical Society of America.

[20]  Shachar Maidenbaum,et al.  Cross-sensory transfer of sensory-motor information: visuomotor learning affects performance on an audiomotor task, using sensory-substitution , 2012, Scientific Reports.

[21]  Isabelle Viaud-Delmon,et al.  From ear to body: the auditory-motor loop in spatial cognition , 2014, Front. Neurosci..

[22]  M. Wanet-Defalque,et al.  Auditory coding of visual patterns for the blind. , 1999, Perception.

[23]  R. Klatzky,et al.  Hand movements: A window into haptic object recognition , 1987, Cognitive Psychology.

[24]  Erik Myin,et al.  Situated perception and sensation in vision and other modalities: a sensorimotor approach , 2009 .

[25]  John Stewart,et al.  Reciprocal Modelling of Active Perception of 2-D Forms in a Simple Tactile-Vision Substitution System , 2004, Minds and Machines.

[26]  Alva Noë,et al.  Neural Plasticity and Consciousness , 2003 .

[27]  A. Clark,et al.  Trading spaces: Computation, representation, and the limits of uninformed learning , 1997, Behavioral and Brain Sciences.

[28]  D. Rayns,et al.  Transverse Tubule Apertures in Mammalian Myocardial Cells: Surface Array , 1967, Science.

[29]  Michael J. Proulx,et al.  Synthetic synaesthesia and sensory substitution , 2010, Consciousness and Cognition.

[30]  Claude Cadoz,et al.  Enaction and Enactive Interfaces : A Handbook of terms , 2007 .

[31]  J. O'Regan,et al.  The Cambridge Handbook of Situated Cognition: Situated Perception and Sensation in Vision and Other Modalities , 2008 .

[32]  Gerard Guarniero,et al.  Tactile Vision: A Personal View , 1977 .

[33]  Mirko Farina.  Neither touch nor vision: sensory substitution as artificial synaesthesia? , 2013 .

[34]  Laurent Renier,et al.  Sensory substitution devices : Creating “artificial synesthesias” , 2013 .

[35]  O. Gapenne.  Kinesthesia and the Construction of Perceptual Objects , 2010 .

[36]  Lea Fleischer,et al.  The Senses Considered As Perceptual Systems , 2016 .

[37]  A. Noë,et al.  A sensorimotor account of vision and visual consciousness. , 2001, The Behavioral and brain sciences.

[38]  Jamie Ward,et al.  Seeing with Sound? Exploring Different Characteristics of a Visual-to-Auditory Sensory Substitution Device , 2011, Perception.

[39]  Daniel D. Hutto.  Knowing What? Radical Versus Conservative Enactivism , 2005 .

[40]  J. Prinz.  Putting the Brakes on Enactive Perception , 2006 .

[41]  Katia Rovira,et al.  Analyse de l'exploration tactile sur support traditionnel chez la personne aveugle et conception de l'interface de lecture Tactos , 2005, Rev. d'Intelligence Artif..

[42]  John Stewart,et al.  Analyse descriptive de trajectories perceptives , 2006, IHM '06.

[43]  F. Vignemont.  A Review of Shaun Gallagher, How the Body Shapes the Mind , 2006 .

[44]  N. Humphrey.  A History of the Mind , 1992 .

[45]  Oscar A. Ramos,et al.  Psychoacoustic tests for the study of human echolocation ability , 1997 .

[46]  J. O'Regan,et al.  Learning to Perceive with a Visuo — Auditory Substitution System: Localisation and Object Recognition with ‘The Voice’ , 2007, Perception.

[47]  F. Paglieri.  Consciousness in Interaction: The role of the natural and social context in shaping consciousness , 2012 .

[48]  Malika Auvray.  Immersion et perception spatiale : l'exemple des dispositifs de substitution sensorielle , 2004 .

[49]  C E Rice,et al.  Human Echo Perception , 1967, Science.

[50]  A. Streri,et al.  Touching for knowing : cognitive psychology of haptic manual perception , 2003 .

[51]  T. Roberts,et al.  Understanding ‘sensorimotor understanding’ , 2010 .

[52]  Katia Rovira,et al.  Learning to recognize shapes with a sensory substitution system: A longitudinal study with 4 non-sighted adolescents , 2010, 2010 IEEE 9th International Conference on Development and Learning.

[53]  P. Stoerig,et al.  Seeing sounds and tingling tongues: Qualia in synaesthesia and sensory substitution , 2006 .

[54]  R H De Jong.  Seeing with sound. , 1977, JAMA.

[55]  Alexander Maye,et al.  Time Scales of Sensorimotor Contingencies , 2012, BICS.

[56]  Donald M. MacKay,et al.  Theoretical Models of Space Perception , 1962 .

[57]  Laurent Renier,et al.  Vertical-horizontal illusion present for sighted but not early blind humans using auditory substitution of vision , 2006, Perception & psychophysics.

[58]  Charles Spence,et al.  The Perception of Space and Form Recognition in a Simulated Environment: The Case of Minimalist Sensory-Substitution Devices , 2007, Perception.

[59]  R P Power,et al.  Echo Perception of Shape and Texture by Sighted Subjects , 1982, Perceptual and motor skills.

[60]  Charles Lenay,et al.  Beyond the internalism/externalism debate: The constitution of the space of perception , 2010, Consciousness and Cognition.

[61]  Anne Bol,et al.  Cross-modal activation of visual cortex during depth perception using auditory substitution of vision , 2005, NeuroImage.

[62]  Charles Lenay,et al.  Dynamics of shape recognition through a minimal visuo-tactile sensory substitution interface , 1999 .

[63]  K. Rovira,et al.  Suppléance perceptive chez l’adolescent aveugle : stratégies individuelles, perception et catégorisation de forme , 2014 .

[64]  P. Bach-y-Rita.  Brain mechanisms in sensory substitution , 1972 .

[65]  E. Rosch,et al.  The Embodied Mind: Cognitive Science and Human Experience , 1993 .

[66]  Gang Zhang,et al.  Quantitative assessment on the cloning efficiencies of lentiviral transfer vectors with a unique clone site , 2012, Scientific Reports.

[67]  Frans A. Bilsen,et al.  Repetition pitch and its implications for hearing theory , 1969 .

[68]  Amir Amedi,et al.  ‘Visual’ Acuity of the Congenitally Blind Using Visual-to-Auditory Sensory Substitution , 2012, PloS one.

[69]  Xabier E. Barandiaran,et al.  Human Neuroscience Hypothesis and Theory Article Learning to Perceive in the Sensorimotor Approach: Piaget's Theory of Equilibration Interpreted Dynamically , 2022 .

[70]  Peter B. L. Meijer,et al.  An experimental system for auditory image representations , 1992, IEEE Transactions on Biomedical Engineering.

[71]  Shachar Maidenbaum,et al.  EyeMusic: Introducing a "visual" colorful experience for the blind using auditory sensory substitution. , 2014, Restorative neurology and neuroscience.

[72]  J. Kiverstein,et al.  Do sensory substitution devices extend the conscious mind? , 2012 .

[73]  Jung-Kyong Kim,et al.  Generalized Learning of Visual-to-auditory Substitution in Sighted Individuals , 2008 .

[74]  P. Bach-y-Rita,et al.  Form perception with a 49-point electrotactile stimulus array on the tongue: a technical note. , 1998, Journal of rehabilitation research and development.

[75]  Peter König,et al.  Sensory Augmentation for the Blind , 2012, Front. Hum. Neurosci..

[76]  P. König,et al.  Beyond sensory substitution—learning the sixth sense , 2005, Journal of neural engineering.

[77]  Thomas D. Wright,et al.  Neuroscience and Biobehavioral Reviews Sensory Substitution as an Artificially Acquired Synaesthesia , 2022 .

[78]  Harald Atmanspacher,et al.  Acategorial states in a reprsentational theory of mental processes , 2010 .

[79]  J. B. Pittenger,et al.  Human Echolocation as a Basic Form of Perception and Action , 1995 .

[80]  Á. Pascual-Leone,et al.  The metamodal organization of the brain. , 2001, Progress in brain research.

[81]  N. Sribunruangrit,et al.  Speed-accuracy tradeoff during performance of a tracking task without visual feedback , 2004, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[82]  M. Auvray,et al.  A Crossmodal Perspective on Sensory Substitution , 2014 .

[83]  Malika Auvray,et al.  Perception With Compensatory Devices: From Sensory Substitution to Sensorimotor Extension , 2009, Cogn. Sci..

[84]  P. Stoerig,et al.  Seeing ‘Where’ through the Ears: Effects of Learning-by-Doing and Long-Term Sensory Deprivation on Localization Based on Image-to-Sound Substitution , 2008, PloS one.

[85]  Mercedes X. Hüg,et al.  Echolocation: An Action-Perception Phenomenon , 2012 .

[86]  Marek McGann,et al.  Perceptual Modalities: Modes of Presentation or Modes of Interaction? , 2010 .

[87]  C. Lenay,et al.  SENSORY SUBSTITUTION: LIMITS AND PERSPECTIVES , 2003 .

[88]  G Guarniero,et al.  Experience of Tactile Vision , 1974, Perception.

[89]  E. D. Di Paolo,et al.  Enaction and Psychology , 2013 .

[90]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.