Reward feedback accelerates motor learning.

Recent findings have demonstrated that reward feedback alone can drive motor learning. However, it is not yet clear whether reward feedback alone can lead to learning when a perturbation is introduced abruptly, or how a reward gradient can modulate learning. In this study, we provide reward feedback that decays continuously with increasing error. We asked whether it is possible to learn an abrupt visuomotor rotation by reward alone, and if the learning process could be modulated by combining reward and sensory feedback and/or by using different reward landscapes. We designed a novel visuomotor learning protocol during which subjects experienced an abruptly introduced rotational perturbation. Subjects received either visual feedback or reward feedback, or a combination of the two. Two different reward landscapes, where the reward decayed either linearly or cubically with distance from the target, were tested. Results demonstrate that it is possible to learn from reward feedback alone and that the combination of reward and sensory feedback accelerates learning. An analysis of the underlying mechanisms reveals that although reward feedback alone does not allow for sensorimotor remapping, it can nonetheless lead to broad generalization, highlighting a dissociation between remapping and generalization. Also, the combination of reward and sensory feedback accelerates learning without compromising sensorimotor remapping. These findings suggest that the use of reward feedback is a promising approach to either supplement or substitute sensory feedback in the development of improved neurorehabilitation techniques. More generally, they point to an important role played by reward in the motor learning process.

[1]  R. C. Oldfield The assessment and analysis of handedness: the Edinburgh inventory. , 1971, Neuropsychologia.

[2]  Maja J. Mataric,et al.  Reward Functions for Accelerated Learning , 1994, ICML.

[3]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[4]  Mark Hallett,et al.  Motor skill learning in Parkinson's disease , 1996, Journal of the Neurological Sciences.

[5]  C Ghez,et al.  Learning of Visuomotor Transformations for Vectorial Planning of Reaching Trajectories , 2000, The Journal of Neuroscience.

[6]  James Hanley,et al.  If we're so different, why do we keep overlapping? When 1 plus 1 doesn't make 2. , 2002, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[7]  O. Hikosaka,et al.  Dopamine Neurons Can Represent Context-Dependent Prediction Error , 2004, Neuron.

[8]  J. O'Doherty,et al.  Reward representations and reward-related learning in the human brain: insights from neuroimaging , 2004, Current Opinion in Neurobiology.

[9]  M. Gluck,et al.  The role of dopamine in cognitive sequence learning: evidence from Parkinson’s disease , 2005, Behavioural Brain Research.

[10]  R. Shadmehr,et al.  Intact ability to learn internal models of arm dynamics in Huntington's disease but not cerebellar degeneration. , 2005, Journal of neurophysiology.

[11]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 2005, IEEE Transactions on Neural Networks.

[12]  J. Krakauer,et al.  An Implicit Plan Overrides an Explicit Strategy during Visuomotor Adaptation , 2006, The Journal of Neuroscience.

[13]  M. Roesch,et al.  Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards , 2007, Nature Neuroscience.

[14]  M. Ernst,et al.  The statistical determinants of adaptation rate in human reaching. , 2008, Journal of vision.

[15]  Paul R. Schrater,et al.  Structure Learning in Human Sequential Decision-Making , 2008, NIPS.

[16]  Colin Camerer,et al.  Neuroeconomics: decision making and the brain , 2008 .

[17]  J. Krakauer,et al.  Explaining savings for visuomotor adaptation: linear time-invariant state-space models are not sufficient. , 2008, Journal of neurophysiology.

[18]  Amir Abbas Zadpoor,et al.  Application of Virtual Environments to Assessment of Human Motor Learning During Reaching Movements , 2009, PRESENCE: Teleoperators and Virtual Environments.

[19]  Daniel M. Wolpert,et al.  Transfer of Dynamic Learning Across Postures , 2009, Journal of neurophysiology.

[20]  Konrad Paul Kording,et al.  Relevance of error: what drives motor adaptation? , 2009, Journal of neurophysiology.

[21]  M. Ghilardi,et al.  Learning and consolidation of visuo-motor adaptation in Parkinson's disease. , 2009, Parkinsonism & related disorders.

[22]  P. Dayan,et al.  States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning , 2010, Neuron.

[23]  Konrad P. Körding,et al.  Uncertainty of Feedback and State Estimation Determines the Speed of Motor Adaptation , 2009, Front. Comput. Neurosci..

[24]  Robert F. Hess,et al.  Spatial scale invariance of the amblyopic global motion deficit , 2010 .

[25]  J. Sanes,et al.  Basal ganglia-dependent processes in recalling learned visual-motor adaptations , 2011, Experimental Brain Research.

[26]  Heidi M. Schambra,et al.  Reward Improves Long-Term Retention of a Motor Memory through Induction of Offline Memory Gains , 2011, Current Biology.

[27]  M. Pessiglione,et al.  Dopamine-dependent reinforcement of motor skill learning: evidence from Gilles de la Tourette syndrome. , 2011, Brain : a journal of neurology.

[28]  Lee Spector,et al.  Evolution of reward functions for reinforcement learning , 2011, GECCO.

[29]  A. Furnham,et al.  A literature review of the anchoring effect , 2011 .

[30]  J. Krakauer,et al.  Human sensorimotor learning: adaptation, skill, and beyond , 2011, Current Opinion in Neurobiology.

[31]  Reza Shadmehr,et al.  Learning from Sensory and Reward Prediction Errors during Motor Adaptation , 2011, PLoS Comput. Biol..

[32]  John E. Schlerf,et al.  Dynamic Modulation of Cerebellar Excitability for Abrupt, But Not Gradual, Visuomotor Adaptation , 2012, The Journal of Neuroscience.

[33]  Raymond J. Delnicki,et al.  Overcoming Motor “Forgetting” Through Reinforcement Of Learned Actions , 2012, The Journal of Neuroscience.

[34]  Michael S Landy,et al.  Motor control is decision-making , 2012, Current Opinion in Neurobiology.

[35]  Mollie K. Marko,et al.  Sensitivity to prediction error in reach adaptation. , 2012, Journal of neurophysiology.

[36]  O. Hikosaka,et al.  Learning to represent reward structure: A key to adapting to complex environments , 2012, Neuroscience Research.

[37]  Alaa A. Ahmed,et al.  Stability limits modulate whole-body motor learning. , 2012, Journal of neurophysiology.

[38]  Helen J. Huang,et al.  Reduction of Metabolic Cost during Motor Learning of Arm Reaching Dynamics , 2012, The Journal of Neuroscience.

[39]  Alaa A. Ahmed,et al.  Learning from the value of your mistakes: evidence for a risk-sensitive process in movement adaptation , 2013, Front. Comput. Neurosci..

[40]  Darwin G. Caldwell,et al.  Reinforcement Learning in Robotics: Applications and Real-World Challenges , 2013, Robotics.

[41]  J. Hsu,et al.  Does the dopamine hypothesis explain schizophrenia? , 2013, Reviews in the neurosciences.

[42]  Allison M Okamura,et al.  Cerebellar motor learning: are environment dynamics more important than error size? , 2013, Journal of neurophysiology.

[43]  Konrad Kording,et al.  Credit Assignment during Movement Reinforcement Learning , 2013, PloS one.

[44]  C. Cepeda,et al.  Dopamine imbalance in Huntington's disease: a mechanism for the lack of behavioral flexibility , 2013, Front. Neurosci..

[45]  W. Schultz Updating dopamine reward signals , 2013, Current Opinion in Neurobiology.

[46]  J. Krakauer,et al.  Explicit and Implicit Contributions to Learning in a Sensorimotor Adaptation Task , 2014, The Journal of Neuroscience.

[47]  Peter Dayan,et al.  When Money Is Not Enough: Awareness, Success, and Variability in Motor Learning , 2014, PloS one.

[48]  Helen J. Huang,et al.  Older adults learn less, but still reduce metabolic cost, during motor adaptation. , 2014, Journal of neurophysiology.