Memory-reliant Post-error Slowing Is Associated with Successful Learning and Fronto-occipital Activity

Negative feedback after an action in a cognitive task can lead to devaluing that action on future trials as well as to more cautious responding when encountering that same choice again. These phenomena have been explored in the past by reinforcement learning theories and cognitive control accounts, respectively. Yet, how cognitive control interacts with value updating to give rise to adequate adaptations under uncertainty is less clear. In this fMRI study, we investigated cognitive control-based behavioral adjustments during a probabilistic reinforcement learning task and studied their influence on performance in a later test phase in which the learned value of items is tested. We provide support for the idea that functionally relevant and memory-reliant behavioral adjustments in the form of post-error slowing during reinforcement learning are associated with test performance. Adjusting response speed after negative feedback was correlated with BOLD activity in right inferior frontal gyrus and bilateral middle occipital cortex during the event of receiving the feedback. Bilateral middle occipital cortex activity overlapped partly with activity reflecting feedback deviance from expectations as measured by unsigned prediction error. These results suggest that cognitive control and feature processing cortical regions interact to implement feedback-congruent adaptations beneficial to learning.

[1]  A. Nobre,et al.  Top-down modulation: bridging selective attention and working memory , 2012, Trends in Cognitive Sciences.

[2]  Simon B. Eickhoff,et al.  A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data , 2005, NeuroImage.

[3]  Michael X. Cohen,et al.  Dorsal Striatal–midbrain Connectivity in Humans Predicts How Reinforcements Are Used to Guide Decisions , 2009, Journal of Cognitive Neuroscience.

[4]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[5]  John J. Foxe,et al.  Avoiding another mistake: Error and posterror neural activity associated with adaptive posterror behavior change , 2007, Cognitive, affective & behavioral neuroscience.

[6]  Timothy Edward John Behrens,et al.  Triangulating a Cognitive Control Network Using Diffusion-Weighted Magnetic Resonance Imaging (MRI) and Functional MRI , 2007, The Journal of Neuroscience.

[7]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[8]  Nathaniel D. Daw,et al.  Trial-by-trial data analysis using computational models , 2011 .

[9]  Michael J. Frank,et al.  By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism , 2004, Science.

[10]  P. Dayan,et al.  Tonic dopamine: opportunity costs and the control of response vigor , 2007, Psychopharmacology.

[11]  W. Schultz Neuronal Reward and Decision Signals: From Theories to Data. , 2015, Physiological reviews.

[12]  Felix Blankenburg,et al.  Maintenance and manipulation of somatosensory information in ventrolateral prefrontal cortex , 2014, Human brain mapping.

[13]  Karl J. Friston,et al.  Influence of Uncertainty and Surprise on Human Corticospinal Excitability during Preparation for Action , 2008, Current Biology.

[14]  R. Dolan,et al.  Priming for self-esteem influences the monitoring of one’s own performance , 2010, Social cognitive and affective neuroscience.

[15]  Josiah R. Boivin,et al.  A Causal Link Between Prediction Errors, Dopamine Neurons and Learning , 2013, Nature Neuroscience.

[16]  Michael J. Frank,et al.  Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning , 2007, Proceedings of the National Academy of Sciences.

[17]  T. Chartrand,et al.  The mind in the middle: A practical guide to priming and automaticity research. , 2000 .

[18]  P. Dayan,et al.  Neural Prediction Errors Reveal a Risk-Sensitive Reinforcement-Learning Process in the Human Brain , 2012, The Journal of Neuroscience.

[19]  M. Gluck,et al.  l-dopa impairs learning, but spares generalization, in Parkinson's disease , 2006, Neuropsychologia.

[20]  M. Reuter,et al.  Genetically Determined Differences in Learning from Errors , 2007, Science.

[21]  Tom Eichele,et al.  Posterior Medial Frontal Cortex Activity Predicts Post-Error Adaptations in Task-Related Visual and Motor Areas , 2011, The Journal of Neuroscience.

[22]  J. Mattingley,et al.  Human medial frontal cortex activity predicts learning from errors. , 2008, Cerebral cortex.

[23]  R. Simons,et al.  To err is autonomic: error-related brain potentials, ANS activity, and post-error compensatory behavior. , 2003, Psychophysiology.

[24]  Jonas Persson,et al.  Context-dependent switching between proactive and reactive working memory control mechanisms in the right inferior frontal gyrus , 2012, NeuroImage.

[25]  Leslie G. Ungerleider,et al.  The Representation of Objects in the Human Occipital and Temporal Cortex , 2000, Journal of Cognitive Neuroscience.

[26]  B. Franke,et al.  Dissociable Effects of Dopamine and Serotonin on Reversal Learning , 2013, Neuron.

[27]  M. Kahana,et al.  Neuronal Activity in the Human Subthalamic Nucleus Encodes Decision Conflict during Action Selection , 2012, The Journal of Neuroscience.

[28]  M. Frank,et al.  Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. , 2009, Nature neuroscience.

[29]  Raymond J. Dolan,et al.  Go and no-go learning in reward and punishment: Interactions between affect and effect , 2012, NeuroImage.

[30]  M. Ullsperger,et al.  Dopamine-Mediated Reinforcement Learning Signals in the Striatum and Ventromedial Prefrontal Cortex Underlie Value-Based Choices , 2011, The Journal of Neuroscience.

[31]  K. R. Ridderinkhof,et al.  The Role of the Medial Frontal Cortex in Cognitive Control , 2004, Science.

[32]  Michael X. Cohen,et al.  Behavioral / Systems / Cognitive Reinforcement Learning Signals Predict Future Decisions , 2007 .

[33]  B. Franke,et al.  Dissociable Effects of Dopamine and Serotonin on Reversal Learning , 2013, Neuron.

[34]  P. Dayan,et al.  Differential Encoding of Losses and Gains in the Human Striatum , 2007, The Journal of Neuroscience.

[35]  Michael X. Cohen,et al.  Striatum-medial prefrontal cortex connectivity predicts developmental changes in reinforcement learning. , 2012, Cerebral cortex.

[36]  M. Ullsperger,et al.  Neurophysiology of performance monitoring and adaptive behavior. , 2014, Physiological reviews.

[37]  R. Turner,et al.  Event-Related fMRI: Characterizing Differential Responses , 1998, NeuroImage.

[38]  M. Delgado,et al.  The role of the striatum in aversive learning and aversive prediction errors , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[39]  Andrea A. Kühn,et al.  Error signals in the subthalamic nucleus are related to post-error slowing in patients with Parkinson's disease , 2014, Cortex.

[40]  M. Sakagami,et al.  Functional role of the ventrolateral prefrontal cortex in decision making , 2007, Current Opinion in Neurobiology.

[41]  John M. Pearson,et al.  Surprise Signals in Anterior Cingulate Cortex: Neuronal Encoding of Unsigned Reward Prediction Errors Driving Adjustment in Behavior , 2011, The Journal of Neuroscience.

[42]  S. Eickhoff,et al.  Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis , 2015, Cognitive, affective & behavioral neuroscience.

[43]  J. Pearce,et al.  A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. , 1980 .

[44]  Jane R. Garrison,et al.  Prediction error in reinforcement learning: A meta-analysis of neuroimaging studies , 2013, Neuroscience & Biobehavioral Reviews.

[45]  M. Ullsperger,et al.  Post-Error Adjustments , 2011, Front. Psychology.

[46]  T. Robbins,et al.  Inhibition and the right inferior frontal cortex: one decade on , 2014, Trends in Cognitive Sciences.

[47]  Matthew L. Dixon,et al.  The lateral prefrontal cortex and complex value-based learning and decision making , 2014, Neuroscience & Biobehavioral Reviews.

[48]  K. R. Ridderinkhof,et al.  Error-related brain potentials are differentially related to awareness of response errors: evidence from an antisaccade task. , 2001, Psychophysiology.

[49]  Michael C. Anderson,et al.  The Prefrontal Cortex Achieves Inhibitory Control by Facilitating Subcortical Motor Pathway Connectivity , 2015, The Journal of Neuroscience.

[50]  E. Wagenmakers,et al.  Testing theories of post-error slowing , 2011, Attention, perception & psychophysics.

[51]  Paul J. Laurienti,et al.  An automated method for neuroanatomic and cytoarchitectonic atlas-based interrogation of fMRI data sets , 2003, NeuroImage.

[52]  Michael J. Frank,et al.  Error-Related Negativity Predicts Reinforcement Learning and Conflict Biases , 2005, Neuron.

[53]  E. Eskandar,et al.  Encoding of Both Positive and Negative Reward Prediction Errors by Neurons of the Primate Lateral Prefrontal Cortex and Caudate Nucleus , 2011, The Journal of Neuroscience.

[54]  Danielle C. Turner,et al.  Association Between Response Inhibition and Working Memory in Adult ADHD: A Link to Right Frontal Cortex Pathology? , 2007, Biological Psychiatry.

[55]  M. Botvinick,et al.  Conflict monitoring and cognitive control. , 2001, Psychological review.

[56]  Jonathan D. Cohen,et al.  Anterior Cingulate Conflict Monitoring and Adjustments in Control , 2004, Science.

[57]  C. Curtis,et al.  Persistent activity in the prefrontal cortex during working memory , 2003, Trends in Cognitive Sciences.

[58]  Franziska M. Korb,et al.  Post-Error Behavioral Adjustments Are Facilitated by Activation and Suppression of Task-Relevant and Task-Irrelevant Information Processing , 2010, The Journal of Neuroscience.

[59]  James F. Cavanagh,et al.  Frontal theta links prediction errors to behavioral adaptation in reinforcement learning , 2010, NeuroImage.

[60]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[61]  Dae-Shik Kim,et al.  Dissociation and convergence of the dorsal and ventral visual working memory streams in the human prefrontal cortex , 2013, NeuroImage.