Speech-to-touch sensory substitution: a 10-decibel improvement in speech-in-noise understanding after a short training

Katarzyna Ciesla (  kasia.j.ciesla@gmail.com ) The Baruch Ivcher Institute for Brain, Cognition & Technology, The Baruch Ivcher School of Psychology, Interdisciplinary Center Herzliya (IDC), Herzliya T. Wolak World Hearing Centre, Institute of Physiology and Pathology of Hearing, Warsaw A. Lorens World Hearing Centre, Institute of Physiology and Pathology of Hearing, Warsaw H. Skarżyński World Hearing Centre, Institute of Physiology and Pathology of Hearing, Warsaw A. Amedi The Baruch Ivcher Institute for Brain, Cognition & Technology, The Baruch Ivcher School of Psychology, Interdisciplinary Center Herzliya (IDC), Herzliya

[1]  Lori L. Holt,et al.  Speech perception under adverse conditions: insights from behavioral, computational, and neuroscience research , 2014, Front. Syst. Neurosci..

[2]  Ron Kupers,et al.  Navigation with a sensory substitution device in congenitally blind individuals , 2011, Neuroreport.

[3]  A. John van Opstal,et al.  The Principle of Inverse Effectiveness in Audiovisual Speech Perception , 2019, bioRxiv.

[4]  Witaya Sungkarat,et al.  Vibrotactile activation of the auditory cortices in deaf versus hearing adults , 2007, Neuroreport.

[5]  Kenneth O. Johnson,et al.  The roles and functions of cutaneous mechanoreceptors , 2001, Current Opinion in Neurobiology.

[6]  Shachar Maidenbaum,et al.  Navigation Using Sensory Substitution in Real and Virtual Mazes , 2015, PloS one.

[7]  J. Schwartz,et al.  Cued Speech Enhances Speech-in-Noise Perception. , 2019, Journal of deaf studies and deaf education.

[8]  Adrian Davis,et al.  Aging and Hearing Health: The Life-course Approach. , 2016, The Gerontologist.

[9]  BENJAMIN WHITE,et al.  Vision Substitution by Tactile Image Projection , 1969, Nature.

[10]  S. S. Stevens.  On the psychophysical law. , 1957, Psychological review.

[11]  S. Levänen,et al.  Vibration-induced auditory-cortex activation in a congenitally deaf adult , 1998, Current Biology.

[12]  Brian A. Lawlor,et al.  Association of Age-Related Hearing Loss With Cognitive Function, Cognitive Impairment, and Dementia: A Systematic Review and Meta-analysis , 2017, JAMA otolaryngology-- head & neck surgery.

[13]  Nadia Bolognini,et al.  Multisensory integration in hemianopia and unilateral spatial neglect: Evidence from the sound induced flash illusion , 2016, Neuropsychologia.

[14]  H Levitt,et al.  Adaptive testing in audiology. , 1978, Scandinavian audiology. Supplementum.

[15]  William M. Stern,et al.  Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex , 2007, Nature Neuroscience.

[16]  Fan-Gang Zeng,et al.  Electro-Tactile Stimulation Enhances Cochlear Implant Speech Recognition in Noise , 2017, Scientific Reports.

[17]  G. Deco,et al.  Multisensory contributions to the perception of vibrotactile events , 2009, Behavioural Brain Research.

[18]  S. Rosen,et al.  Visual Speech Perception in Children With Language Learning Impairments. , 2016, Journal of speech, language, and hearing research : JSLHR.

[19]  N. Holmes.  The law of inverse effectiveness in neurons and behaviour: Multisensory integration versus normal variability , 2007, Neuropsychologia.

[20]  Bruce A. Schneider,et al.  Effects of Senescent Changes in Audition and Cognition on Spoken Language Comprehension , 2010 .

[21]  M. Huber,et al.  Cognition in older adults with severe to profound sensorineural hearing loss compared to peers with normal hearing for age , 2019, International journal of audiology.

[22]  A. D. De Volder,et al.  Hearing, feeling or seeing a beat recruits a supramodal network in the auditory dorsal stream , 2017, The European journal of neuroscience.

[23]  Stefan Van der Stigchel,et al.  Multisensory Stimulation to Improve Low- and Higher-Level Sensory Deficits after Stroke: A Systematic Review , 2015, Neuropsychology Review.

[24]  M. Marschark,et al.  Evidence-Based Practices in Deaf Education , 2018, Oxford Scholarship Online.

[25]  David M. Eagleman,et al.  Using space and time to encode vibrotactile information: toward an estimate of the skin’s achievable throughput , 2015, Experimental Brain Research.

[26]  A. Amedi,et al.  Large-Scale Brain Plasticity Following Blindness and the Use of Sensory Substitution Devices , 2010 .

[27]  B. Stein,et al.  Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. , 1986, Journal of neurophysiology.

[28]  L. Werner,et al.  Development of the Mechanisms Underlying Audiovisual Speech Perception Benefit , 2021, Brain sciences.

[29]  Paula C. Stacey,et al.  Visual Speech Benefit in Clear and Degraded Speech Depends on the Auditory Intelligibility of the Talker and the Number of Background Talkers , 2019, Trends in hearing.

[30]  Ryan A. Stevenson,et al.  Multisensory Integration in Cochlear Implant Recipients , 2017, Ear and hearing.

[31]  Shachar Maidenbaum,et al.  Task Selectivity as a Comprehensive Principle for Brain Organization , 2017, Trends in Cognitive Sciences.

[32]  Shachar Maidenbaum,et al.  The effect of extended sensory range via the EyeCane sensory substitution device on the characteristics of visionless virtual navigation. , 2014, Multisensory research.

[33]  G. Bidelman,et al.  Psychobiological Responses Reveal Audiovisual Noise Differentially Challenges Speech Recognition , 2019, Ear and hearing.

[34]  Amir Amedi,et al.  Immediate improvement of speech-in-noise perception through multisensory stimulation via an auditory to tactile sensory substitution , 2019, Restorative neurology and neuroscience.

[35]  J. Leybaert,et al.  Cued Speech for Enhancing Speech Perception and First Language Development of Children With Cochlear Implants , 2010, Trends in amplification.

[36]  L. Bernstein,et al.  Auditory Perceptual Learning for Speech Perception Can be Enhanced by Audiovisual Training , 2013, Front. Neurosci..

[37]  Amir Amedi,et al.  ‘Visual’ Acuity of the Congenitally Blind Using Visual-to-Auditory Sensory Substitution , 2012, PloS one.

[38]  Tobias Goehring,et al.  Vibro-Tactile Enhancement of Speech Intelligibility in Multi-talker Noise for Simulated Cochlear Implant Listening , 2018, Trends in hearing.

[39]  Sung Hwa Hong,et al.  What Is Temporal Fine Structure and Why Is It Important? , 2014, Korean journal of audiology.

[40]  Frank A. Russo,et al.  Compensatory Plasticity in the Deaf Brain: Effects on Perception of Music , 2014, Brain sciences.

[41]  Antoine J. Shahin,et al.  Visual Enhancement of Relevant Speech in a 'Cocktail Party'. , 2020, Multisensory research.

[42]  B. Moore.  The Role of Temporal Fine Structure Processing in Pitch Perception, Masking, and Speech Perception for Normal-Hearing and Hearing-Impaired People , 2008, Journal of the Association for Research in Otolaryngology.

[43]  Jamie Ward,et al.  Cross-Modal Correspondences Enhance Performance on a Colour-to-Sound Sensory Substitution Device. , 2016, Multisensory research.

[44]  Jonathan Z. Simon,et al.  Robust cortical entrainment to the speech envelope relies on the spectro-temporal fine structure , 2014, NeuroImage.

[45]  L. Bernstein,et al.  Multisensory training can promote or impede visual perceptual learning of speech stimuli: visual-tactile vs. visual-auditory training , 2014, Front. Hum. Neurosci..

[46]  Lema Imam,et al.  Noise-induced hearing loss: a modern epidemic? , 2017, British journal of hospital medicine.

[47]  A. Amedi,et al.  Task-selectivity in the sensory deprived brain and sensory substitution approaches for clinical practice , 2020, Multisensory Perception.

[48]  Matthew H. Davis,et al.  Generalization of perceptual learning of vocoded speech. , 2011, Journal of experimental psychology. Human perception and performance.

[49]  P. Bach-y-Rita.  Tactile Sensory Substitution Studies , 2004, Annals of the New York Academy of Sciences.

[50]  I. Keller,et al.  Improvement of Visual Search After Audiovisual Exploration Training in Hemianopic Patients , 2010, Neurorehabilitation and neural repair.

[51]  Veikko Jousmäki,et al.  Evidence of vibrotactile input to human auditory cortex , 2006, NeuroImage.

[52]  J. Peelle,et al.  Prediction and constraint in audiovisual speech perception , 2015, Cortex.

[53]  Peter B. L. Meijer,et al.  An experimental system for auditory image representations , 1992, IEEE Transactions on Biomedical Engineering.

[54]  M. Ptito,et al.  Spatial Competence and Brain Plasticity in Congenital Blindness via Sensory Substitution Devices , 2020, Frontiers in Neuroscience.

[55]  John F Culling,et al.  Head orientation benefit to speech intelligibility in noise for cochlear implant users and in realistic listening conditions. , 2016, The Journal of the Acoustical Society of America.

[56]  S. Soli,et al.  Development of the Hearing in Noise Test for the measurement of speech reception thresholds in quiet and in noise. , 1994, The Journal of the Acoustical Society of America.

[57]  Shachar Maidenbaum,et al.  EyeMusic: Introducing a "visual" colorful experience for the blind using auditory sensory substitution. , 2014, Restorative neurology and neuroscience.

[58]  Gareth W. Young,et al.  Haptics in Music: The Effects of Vibrotactile Stimulus in Low Frequency Auditory Difference Detection Tasks , 2017, IEEE Transactions on Haptics.

[59]  Richard J. M. van Hoesel,et al.  Audio-Visual Speech Intelligibility Benefits with Bilateral Cochlear Implants when Talker Location Varies , 2014, Journal of the Association for Research in Otolaryngology.

[60]  Quincy M. Samus,et al.  Dementia prevention, intervention, and care , 2017, The Lancet.

[61]  P. Bach-y-Rita,et al.  Clinical Application of Sensory Substitution for Balance Control , 2009 .

[62]  A. Macleod,et al.  Quantifying the contribution of vision to speech perception in noise. , 1987, British journal of audiology.