The relationship between reinforcement and explicit control during visuomotor adaptation

The motor system’s ability to adapt to environmental changes is essential for maintaining accurate movements. Such adaptation recruits several distinct systems: cerebellar sensory-prediction error learning, success-based reinforcement, and explicit control. Although much work has focused on the relationship between cerebellar learning and explicit control, there is little research regarding how reinforcement and explicit control interact. To address this, participants first learnt a 20° visuomotor displacement. After reaching asymptotic performance, binary, hit-or-miss feedback (BF) was introduced either with or without visual feedback, the latter promoting reinforcement. Subsequently, retention was assessed using no-feedback trials, with half of the participants in each group being instructed to stop aiming off target. Although BF led to an increase in retention of the visuomotor displacement, instructing participants to stop re-aiming nullified this effect, suggesting explicit control is critical to BF-based reinforcement. In a second experiment, we prevented the expression or development of explicit control during BF performance, by either constraining participants to a short preparation time (expression) or by introducing the displacement gradually (development). Both manipulations strongly impaired BF performance, suggesting reinforcement requires both recruitment and expression of an explicit component. These results emphasise the pivotal role explicit control plays in reinforcement-based motor learning.

[1]  Maurice A. Smith,et al.  The Decay of Motor Memories Is Independent of Context Change Detection , 2015, PLoS Comput. Biol..

[2]  David M. Huberdeau,et al.  Dual-process decomposition in human sensorimotor adaptation , 2015, Current Opinion in Neurobiology.

[3]  M. Hallett,et al.  Rapid plasticity of human cortical movement representation induced by practice. , 1998, Journal of neurophysiology.

[4]  R. Sun,et al.  The interaction of the explicit and the implicit in skill learning: a dual-process approach. , 2005, Psychological review.

[5]  Welber Marinovic,et al.  Estimating the implicit component of visuomotor rotation learning by constraining movement preparation time , 2017, bioRxiv.

[6]  Ethan R. Buch,et al.  Noninvasive cortical stimulation enhances motor skill acquisition over multiple days through an effect on consolidation , 2009, Proceedings of the National Academy of Sciences.

[7]  Sarah E. Pekny,et al.  Protection and Expression of Human Motor Memories , 2011, The Journal of Neuroscience.

[8]  P. Dayan,et al.  Model-based influences on humans’ choices and striatal prediction errors , 2011, Neuron.

[9]  J. Krakauer,et al.  A computational neuroanatomy for motor control , 2008, Experimental Brain Research.

[10]  P. Viviani,et al.  Pointing errors reflect biases in the perception of the initial hand position. , 1998, Journal of neurophysiology.

[11]  Joseph M. Galea,et al.  Predicting explorative motor learning using decision-making and motor noise , 2017, PLoS Comput. Biol..

[12]  Richard B. Ivry,et al.  Characteristics of Implicit Sensorimotor Adaptation Revealed by Task-irrelevant Clamped Feedback , 2017, Journal of Cognitive Neuroscience.

[13]  O. Donchin,et al.  Awareness of Sensorimotor Adaptation to Visual Rotations of Different Size , 2015, PloS one.

[14]  Nicolas Schweighofer,et al.  Between-Trial Forgetting Due to Interference and Time in Motor Adaptation , 2015, PloS one.

[15]  Jordan A Taylor,et al.  Explicit and Implicit Processes Constitute the Fast and Slow Processes of Sensorimotor Learning , 2015, The Journal of Neuroscience.

[16]  Jeremy C. Rietschel,et al.  Increased reward in ankle robotics training enhances motor control and cortical efficiency in stroke. , 2014, Journal of rehabilitation research and development.

[17]  D. Wolpert,et al.  Effective reinforcement learning following cerebellar damage requires a balance between exploration and motor noise , 2015, Brain : a journal of neurology.

[18]  Reza Shadmehr,et al.  Learning from Sensory and Reward Prediction Errors during Motor Adaptation , 2011, PLoS Comput. Biol..

[19]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[20]  Krista M Bond,et al.  Flexible explicit but rigid implicit learning in a visuomotor adaptation task. , 2015, Journal of neurophysiology.

[21]  S. Wise,et al.  Mechanisms of use-dependent plasticity in the human motor cortex. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[22]  J. Galea,et al.  The effects of reward and punishment on motor skill learning , 2018, Current Opinion in Behavioral Sciences.

[23]  Y. Kojima,et al.  Selective reward affects the rate of saccade adaptation , 2017, Neuroscience.

[24]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[25]  J. Rothwell,et al.  Reward and punishment enhance motor adaptation in stroke , 2017, Journal of Neurology, Neurosurgery, and Psychiatry.

[26]  Hiroaki Gomi,et al.  Multiple Motor Learning Strategies in Visuomotor Rotation , 2010, PloS one.

[27]  John W Krakauer,et al.  Motor learning and consolidation: the case of visuomotor rotation. , 2009, Advances in experimental medicine and biology.

[28]  J. Krakauer,et al.  An Implicit Plan Overrides an Explicit Strategy during Visuomotor Adaptation , 2006, The Journal of Neuroscience.

[29]  Stephan Riek,et al.  Savings for visuomotor adaptation require prior history of error, not prior repetition of successful actions. , 2016, Journal of neurophysiology.

[30]  Peter Dayan,et al.  Bonsai Trees in Your Head: How the Pavlovian System Sculpts Goal-Directed Choices by Pruning Decision Trees , 2012, PLoS Comput. Biol..

[31]  P. Dayan,et al.  States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning , 2010, Neuron.

[32]  J. Krakauer,et al.  Explicit and Implicit Contributions to Learning in a Sensorimotor Adaptation Task , 2014, The Journal of Neuroscience.

[33]  Michael J. Frank,et al.  Faculty Opinions recommendation of States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. , 2010 .

[34]  Vincent S. Huang,et al.  Rethinking Motor Learning and Savings in Adaptation Paradigms: Model-Free Memory for Successful Actions Combines with Internal Models , 2011, Neuron.

[35]  Dylan A. Simon,et al.  Neural Correlates of Forward Planning in a Spatial Decision Task in Humans , 2011, The Journal of Neuroscience.

[36]  R. Shadmehr,et al.  Interacting Adaptive Processes with Different Timescales Underlie Short-Term Motor Learning , 2006, PLoS biology.

[37]  Rachael D. Seidler,et al.  Contributions of Spatial Working Memory to Visuomotor Learning , 2010, Journal of Cognitive Neuroscience.

[38]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[39]  Philippe Lefèvre,et al.  Formation of model-free motor memories during motor adaptation depends on perturbation schedule. , 2015, Journal of neurophysiology.

[40]  D. Wolpert,et al.  Internal models in the cerebellum , 1998, Trends in Cognitive Sciences.

[41]  A. Haith,et al.  Unlearning versus savings in visuomotor adaptation: comparing effects of washout, passage of time, and removal of errors on motor memory , 2013, Front. Hum. Neurosci..

[42]  P. Celnik,et al.  Dissociating the roles of the cerebellum and motor cortex during adaptive learning: the motor cortex retains what the cerebellum learns. , 2011, Cerebral cortex.

[43]  Nathaniel D. Daw,et al.  Cognitive Control Predicts Use of Model-based Reinforcement Learning , 2014, Journal of Cognitive Neuroscience.

[44]  Matthew J. Crossley,et al.  Savings upon Re-Aiming in Visuomotor Adaptation , 2015, The Journal of Neuroscience.

[45]  Reza Shadmehr,et al.  Dissociable effects of the implicit and explicit memory systems on learning control of reaching , 2006, Experimental Brain Research.

[46]  J. Rothwell,et al.  The dissociable effects of punishment and reward on motor learning , 2015, Nature Neuroscience.

[47]  J. Galea,et al.  Individual differences in explicit and implicit visuomotor learning and working memory capacity , 2016, Scientific Reports.

[48]  A. Markman,et al.  The Curse of Planning: Dissecting Multiple Reinforcement-Learning Systems by Taxing the Central Executive , 2013 .

[49]  Peter Dayan,et al.  When Money Is Not Enough: Awareness, Success, and Variability in Motor Learning , 2014, PloS one.

[50]  David M. Huberdeau,et al.  The Influence of Movement Preparation Time on the Expression of Visuomotor Learning and Savings , 2015, The Journal of Neuroscience.

[51]  P. Dayan,et al.  Mapping value based planning and extensively trained choice in the human brain , 2012, Nature Neuroscience.

[52]  Richard S. Sutton,et al.  Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming , 1990, ML.

[53]  Raymond J. Delnicki,et al.  Overcoming Motor “Forgetting” Through Reinforcement Of Learned Actions , 2012, The Journal of Neuroscience.

[54]  Philippe Lefèvre,et al.  Overlap of movement planning and movement execution reduces reaction time by up to 100ms , 2016, bioRxiv.

[55]  Alborz Geramifard,et al.  Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping , 2008, UAI.

[56]  A. Haith,et al.  Model-based and model-free mechanisms of human motor learning. , 2013, Advances in experimental medicine and biology.

[57]  J. Flanagan,et al.  Relation between reaction time and reach errors during visuomotor adaptation , 2011, Behavioural Brain Research.

[58]  R. Ivry,et al.  Cerebellar and prefrontal cortex contributions to adaptation, strategies, and reinforcement learning. , 2014, Progress in brain research.

[59]  Richard B. Ivry,et al.  Flexible Cognitive Strategies during Motor Learning , 2011, PLoS Comput. Biol..

[60]  J. Krakauer,et al.  Sensory prediction errors drive cerebellum-dependent adaptation of reaching. , 2007, Journal of neurophysiology.

[61]  Peter Holland,et al.  Contribution of explicit processes to reinforcement-based motor learning. , 2018, Journal of neurophysiology.

[62]  B. Balleine,et al.  A specific role for posterior dorsolateral striatum in human habit learning , 2009, The European journal of neuroscience.

[63]  Stephen G Lisberger,et al.  Role of Plasticity at Different Sites across the Time Course of Cerebellar Motor Learning , 2014, The Journal of Neuroscience.

[64]  Daniel M. Wolpert,et al.  Forward Models for Physiological Motor Control , 1996, Neural Networks.

[65]  A. Haith,et al.  Independence of Movement Preparation and Movement Initiation , 2016, The Journal of Neuroscience.