Substituting auditory for visual feedback to adapt to altered dynamic and kinematic environments during reaching

The arm movement control system often relies on visual feedback to drive motor adaptation and to help specify desired trajectories. Here we studied whether kinematic errors that were indicated with auditory feedback could be used to control reaching in a way comparable with when vision was available. We randomized twenty healthy adult subjects to receive either visual or auditory feedback of their movement trajectory error with respect to a line as they performed timed reaching movements while holding a robotic joystick. We delivered auditory feedback using spatialized pink noise, the loudness and location of which reflected kinematic error. After a baseline period, we unexpectedly perturbed the reaching trajectories using a perpendicular viscous force field applied by the joystick. Subjects adapted to the force field as well with auditory feedback as they did with visual feedback and exhibited comparable after effects when the force field was removed. When we changed the reference trajectory to be a trapezoid instead of a line, subjects shifted their trajectories by about the same amount with either auditory or visual feedback of error. These results indicate that arm motor networks can readily incorporate auditory feedback to alter internal models and desired trajectories, a finding with implications for the organization of the arm motor control adaptation system as well as sensory substitution and motor training technologies.

[1]  BENJAMIN WHITE,et al.  Vision Substitution by Tactile Image Projection , 1969, Nature.

[2]  Juhani Hyvärinen,et al.  Distribution of visual and somatic functions in the parietal associative area 7 of the monkey , 1979, Brain Research.

[3]  F A Mussa-Ivaldi,et al.  Adaptive representation of dynamics during learning of a motor task , 1994, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[4]  Michael I. Jordan,et al.  Computational models of sensorimotor integration , 1997 .

[5]  Pietro G. Morasso,et al.  Self-Organization, Computational Maps, and Motor Control , 1997 .

[6]  Reza Shadmehr,et al.  Learning of action through adaptive combination of motor primitives , 2000, Nature.

[7]  R A Scheidt,et al.  Persistence of motor adaptation during constrained, multi-joint, arm movements. , 2000, Journal of neurophysiology.

[8]  Alexandre Pouget,et al.  A computational perspective on the neural basis of multisensory spatial representations , 2002, Nature Reviews Neuroscience.

[9]  P. Bach-y-Rita,et al.  Sensory substitution and the human–machine interface , 2003, Trends in Cognitive Sciences.

[10]  T. Hackett,et al.  Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[11]  Ankoor S. Shah,et al.  Auditory Cortical Neurons Respond to Somatosensory Stimulation , 2003, The Journal of Neuroscience.

[12]  Michael I. Jordan,et al.  Are arm trajectories planned in kinematic or dynamic coordinates? An adaptation study , 1995, Experimental Brain Research.

[13]  J. Flanagan,et al.  Learning and recall of incremental kinematic and dynamic sensorimotor transformations , 2005, Experimental Brain Research.

[14]  Ferdinando A Mussa-Ivaldi,et al.  Interaction of visual and proprioceptive feedback during adaptation of human reaching movements. , 2005, Journal of neurophysiology.

[15]  Davide Rocchesso,et al.  Continuous sonic feedback from a rolling ball , 2005, IEEE MultiMedia.

[16]  K. Fujii,et al.  Visualization for the analysis of fluid motion , 2005, J. Vis..

[17]  David J Reinkensmeyer,et al.  Effect of muscle fatigue on internal model formation and retention during reaching with the arm. , 2006, Journal of applied physiology.

[18]  J. O'Regan,et al.  Learning to Perceive with a Visuo — Auditory Substitution System: Localisation and Object Recognition with ‘The Voice’ , 2007, Perception.

[19]  Kurt A. Thoroughman,et al.  Divided attention impairs human motor adaptation but not feedback control. , 2007, Journal of neurophysiology.

[20]  M. Ernst,et al.  The statistical determinants of adaptation rate in human reaching. , 2008, Journal of vision.

[21]  J. Contreras-Vidal,et al.  Adaptation of sound localization induced by rotated visual feedback in reaching movements , 2009, Experimental Brain Research.

[22]  Dennis M Levi,et al.  What limits performance in the amblyopic visual system: seeing signals in noise with an amblyopic brain. , 2008, Journal of vision.

[23]  P. Stoerig,et al.  Seeing ‘Where’ through the Ears: Effects of Learning-by-Doing and Long-Term Sensory Deprivation on Localization Based on Image-to-Sound Substitution , 2008, PloS one.

[24]  David J. Reinkensmeyer,et al.  Using Sound feedback to counteract visual distractor during robot-assisted movement training , 2009, 2009 IEEE International Workshop on Haptic Audio visual Environments and Games.

[25]  P. Lindberg,et al.  Effect of auditory feedback differs according to side of hemiparesis: a comparative pilot study , 2009, Journal of NeuroEngineering and Rehabilitation.

[26]  D. Reinkensmeyer,et al.  Effect of visual distraction and auditory feedback on patient effort during robot-assisted movement training after stroke , 2011, Journal of NeuroEngineering and Rehabilitation.

[27]  G. Thielman Rehabilitation of Reaching Poststroke: A Randomized Pilot Investigation of Tactile Versus Auditory Feedback for Trunk Control , 2010, Journal of neurologic physical therapy : JNPT.

[28]  Sarah E. Criscimagna-Hemminger,et al.  Size of error affects cerebellar contributions to motor learning. , 2010, Journal of neurophysiology.

[29]  F. Lepore,et al.  Sensory rehabilitation in the plastic brain. , 2011, Progress in brain research.

[30]  Gerdienke B. Prange,et al.  The role of visual feedback in conventional therapy and future research , 2011, 2011 International Conference on Virtual Rehabilitation.

[31]  Jamie Ward,et al.  Seeing with Sound? Exploring Different Characteristics of a Visual-to-Auditory Sensory Substitution Device , 2011, Perception.

[32]  Katja Fiehler,et al.  Testing the limits of optimal integration of visual and proprioceptive information of path trajectory , 2011, Experimental Brain Research.

[33]  Sarah E. Criscimagna-Hemminger,et al.  Contributions of the motor cortex to adaptive control of reaching depend on the perturbation schedule. , 2011, Cerebral cortex.

[34]  A. Amedi,et al.  The brain as a flexible task machine: implications for visual rehabilitation using noninvasive vs. invasive approaches. , 2012, Current opinion in neurology.