Modulation of feedback related activity in the rostral anterior cingulate cortex during trial and error exploration

The present functional magnetic resonance imaging (fMRI) experiment aims at clarifying the role of the rostral anterior cingulate cortex (rACC) in the evaluation of feedback in a deterministic environment. We tested, in particular, the response of the rACC to the detection of different types of feedback, and to varying levels of outcome expectancy. We used a problem-solving task in which subjects had to discover, in successive trials, which one of the four presented stimuli was associated with a positive feedback, the other ones being associated with error feedback. In this task, two periods alternated: 1. an exploratory period in which error feedback indicated to adapt the following response appropriately (and continue to explore), and first positive feedback indicated to change strategy (i.e. to shift from explorative to exploitative behavior), and 2. an exploitative period in which subjects had to repeat the correct choice. The rACC is recruited in the exploratory period during the analysis of both error and first correct positive feedback. In addition, the rACC activity was modulated by positive reward prediction error values (i.e. the difference between obtained and expected feedback). Altogether, these results reveal the critical role of the rACC in the evaluation of salient feedback for learning optimal strategies.

[1]  Timothy E. J. Behrens,et al.  Counterfactual Choice and Learning in a Neural Network Centered on Human Lateral Frontopolar Cortex , 2011, PLoS biology.

[2]  Jonathan D. Cohen,et al.  Computational roles for dopamine in behavioural control , 2004, Nature.

[3]  M. Petrides,et al.  Selective involvement of the mid-dorsolateral prefrontal cortex in the coding of the serial order of visual stimuli in working memory , 2007, Proceedings of the National Academy of Sciences.

[4]  Keiji Tanaka,et al.  Medial prefrontal cell activity signaling prediction errors of action values , 2007, Nature Neuroscience.

[5]  D. Durstewitz,et al.  The ability of the mesocortical dopamine system to operate in distinct temporal modes , 2007, Psychopharmacology.

[6]  Andrew Caplin,et al.  Axiomatic Methods, Dopamine and Reward Prediction Error This Review Comes from a Themed Issue on Cognitive Neuroscience Edited Advantages of the Axiomatic Approach , 2022 .

[7]  P. Apicella,et al.  Tonically active neurons in the striatum differentiate between delivery and omission of expected reward in a probabilistic task context , 2009, The European journal of neuroscience.

[8]  R W Cox,et al.  Real‐time 3D image registration for functional MRI , 1999, Magnetic resonance in medicine.

[9]  D. Durstewitz,et al.  The Dual-State Theory of Prefrontal Cortex Dopamine Function with Relevance to Catechol-O-Methyltransferase Genotypes and Schizophrenia , 2008, Biological Psychiatry.

[10]  Timothy Edward John Behrens,et al.  Effort-Based Cost–Benefit Valuation and the Human Brain , 2009, The Journal of Neuroscience.

[11]  P. Glimcher,et al.  Action and Outcome Encoding in the Primate Caudate Nucleus , 2007, The Journal of Neuroscience.

[12]  Michael Petrides,et al.  Response selection versus feedback analysis in conditional visuo-motor learning , 2012, NeuroImage.

[13]  M. Rushworth,et al.  Behavioral / Systems / Cognitive Connectivity-Based Parcellation of Human Cingulate Cortex and Its Relation to Functional Specialization , 2008 .

[14]  P. Goldman-Rakic,et al.  Modulation of Dorsolateral Prefrontal Delay Activity during Self-Organized Behavior , 2006, The Journal of Neuroscience.

[15]  O. Hikosaka,et al.  Switching from automatic to controlled action by monkey medial frontal cortex , 2007, Nature Neuroscience.

[16]  Karl J. Friston,et al.  Analysis of fMRI Time-Series Revisited , 1995, NeuroImage.

[17]  M. Kawato,et al.  Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. , 2006, Journal of neurophysiology.

[18]  P. Apicella,et al.  The Role of Striatal Tonically Active Neurons in Reward Prediction Error Signaling during Instrumental Task Performance , 2011, The Journal of Neuroscience.

[19]  Emmanuel Procyk,et al.  Coordination of High Gamma Activity in Anterior Cingulate and Lateral Prefrontal Cortical Areas during Adaptation , 2011, The Journal of Neuroscience.

[20]  Michael Petrides,et al.  Dissociable roles of the posterior parietal and the prefrontal cortex in manipulation and monitoring processes , 2007, Proceedings of the National Academy of Sciences.

[21]  G. Pagnoni,et al.  Human Striatal Responses to Monetary Reward Depend On Saliency , 2004, Neuron.

[22]  Ryan K. Jessup,et al.  Error Effects in Anterior Cingulate Cortex Reverse when Error Likelihood Is High , 2010, The Journal of Neuroscience.

[23]  Sean M. Polyn,et al.  Beyond mind-reading: multi-voxel pattern analysis of fMRI data , 2006, Trends in Cognitive Sciences.

[24]  E. Procyk,et al.  Anterior cingulate error‐related activity is modulated by predicted reward , 2005, The European journal of neuroscience.

[25]  M. Petrides,et al.  Specialized systems for the processing of mnemonic information within the primate frontal cortex. , 1996, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[26]  Timothy Edward John Behrens,et al.  How Green Is the Grass on the Other Side? Frontopolar Cortex and the Evidence in Favor of Alternative Courses of Action , 2009, Neuron.

[27]  M Petrides,et al.  Monitoring of selections of visual stimuli and the primate frontal cortex , 1991, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[28]  Michael X. Cohen,et al.  Cortical electrophysiological network dynamics of feedback learning , 2011, Trends in Cognitive Sciences.

[29]  S. Ishii,et al.  Resolution of Uncertainty in Prefrontal Cortex , 2006, Neuron.

[30]  W. Schultz,et al.  Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli , 1996, Nature.

[31]  P. Goldman-Rakic,et al.  Selective D2 Receptor Actions on the Functional Circuitry of Working Memory , 2004, Science.

[32]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[33]  Clay B. Holroyd,et al.  Errors in reward prediction are re£ected in the event-related brain potential , 2003 .

[34]  B. Hayden,et al.  Neurons in Anterior Cingulate Cortex Multiplex Information about Reward and Action , 2010, The Journal of Neuroscience.

[35]  Timothy E. J. Behrens,et al.  Choice, uncertainty and value in prefrontal and cingulate cortex , 2008, Nature Neuroscience.

[36]  M. Kimura,et al.  Neuronal basis for evaluating selected action in the primate striatum , 2011, The European journal of neuroscience.

[37]  S. Dehaene,et al.  Dynamics of prefrontal and cingulate activity during a reward-based logical deduction task. , 2006, Cerebral cortex.

[38]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[39]  K. A. Hadland,et al.  The anterior cingulate and reward-guided selection of actions. , 2003, Journal of neurophysiology.

[40]  A. Rangel,et al.  Dissociating valuation and saliency signals during decision-making. , 2011, Cerebral cortex.

[41]  M Petrides,et al.  Impairments on nonspatial self-ordered and externally ordered working memory tasks after lesions of the mid-dorsal part of the lateral frontal cortex in the monkey , 1995, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[42]  E. Procyk,et al.  Behavioral Shifts and Action Valuation in the Anterior Cingulate Cortex , 2008, Neuron.

[43]  Timothy E. J. Behrens,et al.  Optimal decision making and the anterior cingulate cortex , 2006, Nature Neuroscience.

[44]  Erin L. Rich,et al.  Challenges of Interpreting Frontal Neurons during Value-Based Decision-Making , 2011, Front. Neurosci..

[45]  D. Collins,et al.  Automatic 3D Intersubject Registration of MR Volumetric Data in Standardized Talairach Space , 1994, Journal of computer assisted tomography.

[46]  François Mauguière,et al.  Brain responses to success and failure: Direct recordings from human cerebral cortex , 2010, Human brain mapping.

[47]  John M. Pearson,et al.  Surprise Signals in Anterior Cingulate Cortex: Neuronal Encoding of Unsigned Reward Prediction Errors Driving Adjustment in Behavior , 2011, The Journal of Neuroscience.

[48]  Joshua W. Brown,et al.  Performance Monitoring by the Anterior Cingulate Cortex During Saccade Countermanding , 2003, Science.

[49]  Clay B. Holroyd,et al.  The feedback-related negativity reflects the binary evaluation of good versus bad outcomes , 2006, Biological Psychology.

[50]  S. Quartz,et al.  Human Insula Activation Reflects Risk Prediction Errors As Well As Risk , 2008, The Journal of Neuroscience.

[51]  E. Procyk,et al.  Expectations, gains, and losses in the anterior cingulate cortex , 2007, Cognitive, affective & behavioral neuroscience.

[52]  David Goodman,et al.  Performance Monitoring in the Anterior Cingulate is Not All Error Related: Expectancy Deviation and the Representation of Action-Outcome Associations , 2007, Journal of Cognitive Neuroscience.

[53]  M. Walton,et al.  Interactions between decision making and performance monitoring within prefrontal cortex , 2004, Nature Neuroscience.

[54]  N. Bargalló,et al.  Anterior cingulate and paracingulate sulci morphology in patients with schizophrenia , 2010, Schizophrenia Research.

[55]  Andrea Brovelli,et al.  Understanding the neural computations of arbitrary visuomotor learning through fMRI and associative learning theory. , 2008, Cerebral cortex.

[56]  E. Bullmore,et al.  Statistical methods of estimation and inference for functional MR image analysis , 1996, Magnetic resonance in medicine.

[57]  D. Pandya,et al.  Fiber system linking the mid‐dorsolateral frontal cortex with the retrosplenial/presubicular region in the rhesus monkey , 1999, The Journal of comparative neurology.

[58]  Zhong-Lin Lu,et al.  Neural correlates of risk prediction error during reinforcement learning in humans , 2009, NeuroImage.

[59]  M. Petrides,et al.  Does the cingulate cortex contribute to spatial conditional associative learning in the rat? , 2009, Hippocampus.

[60]  Karl J. Friston,et al.  A unified statistical approach for determining significant signals in images of cerebral activation , 1996, Human brain mapping.

[61]  Timothy E. J. Behrens,et al.  Review Frontal Cortex and Reward-guided Learning and Decision-making Figure 1. Frontal Brain Regions in the Macaque Involved in Reward-guided Learning and Decision-making Finer Grained Anatomical Divisions with Frontal Cortical Systems for Reward-guided Behavior , 2022 .

[62]  Karl J. Friston,et al.  Conjunction revisited , 2005, NeuroImage.

[63]  Michael X. Cohen,et al.  Reward expectation modulates feedback-related negativity and EEG spectra , 2007, NeuroImage.

[64]  S. Kennerley,et al.  Evaluating choices by single neurons in the frontal lobe: outcome value encoded across multiple decision variables , 2009, The European journal of neuroscience.

[65]  Alan C. Evans,et al.  A General Statistical Analysis for fMRI Data , 2000, NeuroImage.

[66]  Karl Zilles,et al.  Cytology and receptor architecture of human anterior cingulate cortex , 2008, The Journal of comparative neurology.

[67]  Deepak N. Pandya,et al.  The Frontal Cortex , 2012 .

[68]  Problem solving and logical reasoning in the macaque monkey , 1996, Behavioural Brain Research.

[69]  Ryan K. Jessup,et al.  Human Dorsal Striatal Activity during Choice Discriminates Reinforcement Learning Behavior from the Gambler's Fallacy , 2011, The Journal of Neuroscience.

[70]  R. Baker,et al.  When is an error not a prediction error? An electrophysiological investigation , 2009, Cognitive, affective & behavioral neuroscience.

[71]  Benjamin Y. Hayden,et al.  Human Neuroscience , 2022 .

[72]  RP Dum,et al.  The origin of corticospinal projections from the premotor areas in the frontal lobe , 1991, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[73]  J. Tanji,et al.  Role for cingulate motor area cells in voluntary movement selection based on reward. , 1998, Science.

[74]  M. Petrides,et al.  Differential activation of the human orbital, mid-ventrolateral, and mid-dorsolateral prefrontal cortex during the processing of visual stimuli , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[75]  Clay B. Holroyd,et al.  The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. , 2002, Psychological review.

[76]  P. Glimcher,et al.  Testing the Reward Prediction Error Hypothesis with an Axiomatic Model , 2010, The Journal of Neuroscience.

[77]  Emmanuel Procyk,et al.  Frontal Feedback-Related Potentials in Nonhuman Primates: Modulation during Learning and under Haloperidol , 2009, The Journal of Neuroscience.

[78]  James V. Haxby,et al.  Multivariate pattern analysis of fMRI: The early beginnings , 2012, NeuroImage.

[79]  John M. Pearson,et al.  Neuronal basis of sequential foraging decisions in a patchy environment , 2011, Nature Neuroscience.

[80]  Daeyeol Lee,et al.  Ubiquity and Specificity of Reinforcement Signals throughout the Human Brain , 2011, Neuron.

[81]  E. Procyk,et al.  Anterior cingulate activity during routine and non-routine sequential behaviors in macaques , 2000, Nature Neuroscience.

[82]  Timothy E. J. Behrens,et al.  Frontal Cortex Subregions Play Distinct Roles in Choices between Actions and Stimuli , 2008, The Journal of Neuroscience.

[83]  Mitsuo Kawato,et al.  Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning , 2006, Neural Networks.

[84]  Steven Laureys,et al.  Cytology and functionally correlated circuits of human posterior cingulate areas , 2006, NeuroImage.

[85]  E. Procyk,et al.  Reward encoding in the monkey anterior cingulate cortex. , 2006, Cerebral cortex.

[86]  Alan C. Evans,et al.  Dissociation of human mid-dorsolateral from posterior dorsolateral frontal cortex in memory processing. , 1993, Proceedings of the National Academy of Sciences of the United States of America.

[87]  Daeyeol Lee,et al.  Behavioral and Neural Changes after Gains and Losses of Conditioned Reinforcers , 2009, The Journal of Neuroscience.