The Feedback-related Negativity Codes Components of Abstract Inference during Reward-based Decision-making

Behavioral control is influenced not only by learning from the choices made and the rewards obtained but also by “what might have happened,” that is, inference about unchosen options and their fictive outcomes. Substantial progress has been made in understanding the neural signatures of direct learning from choices that are actually made and their associated rewards via reward prediction errors (RPEs). However, electrophysiological correlates of abstract inference in decision-making are less clear. One seminal theory suggests that the so-called feedback-related negativity (FRN), an ERP peaking 200–300 msec after a feedback stimulus at frontocentral sites of the scalp, codes RPEs. Hitherto, the FRN has been predominantly related to a so-called “model-free” RPE: The difference between the observed outcome and what had been expected. Here, by means of computational modeling of choice behavior, we show that individuals employ abstract, “double-update” inference on the task structure by concurrently tracking values of chosen stimuli (associated with observed outcomes) and unchosen stimuli (linked to fictive outcomes). In a parametric analysis, model-free RPEs as well as their modification because of abstract inference were regressed against single-trial FRN amplitudes. We demonstrate that components related to abstract inference uniquely explain variance in the FRN beyond model-free RPEs. These findings advance our understanding of the FRN and its role in behavioral adaptation. This might further the investigation of disturbed abstract inference, as proposed, for example, for psychiatric disorders, and its underlying neural correlates.

[1]  Raymond J. Dolan,et al.  An MEG signature corresponding to an axiomatic model of reward prediction error , 2012, NeuroImage.

[2]  Andreas Heinz,et al.  Reinforcement Learning and Dopamine in Schizophrenia: Dimensions of Symptoms or Specific Features of a Disease Group? , 2013, Front. Psychiatry.

[3]  Simon Hong,et al.  A pallidus-habenula-dopamine pathway signals inferred stimulus values. , 2010, Journal of neurophysiology.

[4]  Timothy E. J. Behrens,et al.  Counterfactual Choice and Learning in a Neural Network Centered on Human Lateral Frontopolar Cortex , 2011, PLoS biology.

[5]  Hauke R. Heekeren,et al.  Temporal dynamics of prediction error processing during reward-based decision making , 2010, NeuroImage.

[6]  N. Daw,et al.  Signals in Human Striatum Are Appropriate for Policy Update Rather than Value Prediction , 2011, The Journal of Neuroscience.

[7]  N. Daw,et al.  Differential roles of human striatum and amygdala in associative learning , 2011, Nature Neuroscience.

[8]  D. Lehmann,et al.  Reference-free identification of components of checkerboard-evoked multichannel potential fields. , 1980, Electroencephalography and clinical neurophysiology.

[9]  D. Shohamy,et al.  Integrating Memories in the Human Brain: Hippocampal-Midbrain Encoding of Overlapping Events , 2008, Neuron.

[10]  Lilianne R. Mujica-Parodi,et al.  Ventral striatal and medial prefrontal BOLD activation is correlated with reward-related electrocortical activity: A combined ERP and fMRI study , 2011, NeuroImage.

[11]  P. Dayan,et al.  States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning , 2010, Neuron.

[12]  P. Dayan,et al.  Disorders of compulsivity: a common bias towards learning habits , 2014, Molecular Psychiatry.

[13]  Alexander M. Nitsch,et al.  A Single-Trial Estimation of the Feedback-Related Negativity and Its Relation to BOLD Responses in a Time-Estimation Task , 2014, The Journal of Neuroscience.

[14]  M. Frank,et al.  Frontal theta as a mechanism for cognitive control , 2014, Trends in Cognitive Sciences.

[15]  James F. Cavanagh,et al.  Frontal theta links prediction errors to behavioral adaptation in reinforcement learning , 2010, NeuroImage.

[16]  T. Robbins,et al.  Goal-directed learning and obsessive–compulsive disorder , 2014, Philosophical Transactions of the Royal Society B: Biological Sciences.

[17]  Michael X. Cohen,et al.  Cortical electrophysiological network dynamics of feedback learning , 2011, Trends in Cognitive Sciences.

[18]  N. Daw,et al.  The ubiquity of model-based reinforcement learning , 2012, Current Opinion in Neurobiology.

[19]  Timothy Edward John Behrens,et al.  How Green Is the Grass on the Other Side? Frontopolar Cortex and the Evidence in Favor of Alternative Courses of Action , 2009, Neuron.

[20]  Shinsuke Shimojo,et al.  Neural Computations Underlying Arbitration between Model-Based and Model-free Learning , 2013, Neuron.

[21]  Deborah Talmi,et al.  The Feedback-Related Negativity Signals Salience Prediction Errors, Not Reward Prediction Errors , 2013, The Journal of Neuroscience.

[22]  Roshan Cools,et al.  Feedback-related Negativity Codes Prediction Error but Not Behavioral Adjustment during Probabilistic Reversal Learning , 2011, Journal of Cognitive Neuroscience.

[23]  Karl J. Friston,et al.  Bayesian model selection for group studies , 2009, NeuroImage.

[24]  A. Villringer,et al.  The interaction of acute and chronic stress impairs model-based behavioral control , 2015, Psychoneuroendocrinology.

[25]  P. Dayan,et al.  Model-based influences on humans’ choices and striatal prediction errors , 2011, Neuron.

[26]  R. Dolan,et al.  Dopamine Enhances Model-Based over Model-Free Choice Behavior , 2012, Neuron.

[27]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[28]  Joshua W. Brown,et al.  Medial prefrontal cortex as an action-outcome predictor , 2011, Nature Neuroscience.

[29]  Shu-Chen Li,et al.  Of goals and habits: age-related and individual differences in goal-directed decision-making , 2013, Front. Neurosci..

[30]  Anna Weinberg,et al.  Event‐related potential activity in the basal ganglia differentiates rewards from nonrewards: Temporospatial principal components analysis and source localization of the feedback negativity , 2011, Human brain mapping.

[31]  Timothy E. J. Behrens,et al.  Review Frontal Cortex and Reward-guided Learning and Decision-making Figure 1. Frontal Brain Regions in the Macaque Involved in Reward-guided Learning and Decision-making Finer Grained Anatomical Divisions with Frontal Cortical Systems for Reward-guided Behavior , 2022 .

[32]  Thomas H. B. FitzGerald,et al.  Disruption of Dorsolateral Prefrontal Cortex Decreases Model-Based in Favor of Model-free Control in Humans , 2013, Neuron.

[33]  G. Schoenbaum,et al.  Orbitofrontal activation restores insight lost after cocaine use , 2014, Nature Neuroscience.

[34]  Raymond J. Dolan,et al.  Disentangling the Roles of Approach, Activation and Valence in Instrumental and Pavlovian Responding , 2011, PLoS Comput. Biol..

[35]  L. Deserno,et al.  Risk Factors for Addiction and Their Association with Model-Based Behavioral Control , 2016, Front. Behav. Neurosci..

[36]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[37]  Peter Dayan,et al.  Bonsai Trees in Your Head: How the Pavlovian System Sculpts Goal-Directed Choices by Pruning Decision Trees , 2012, PLoS Comput. Biol..

[38]  Tobias U. Hauser,et al.  The feedback-related negativity (FRN) revisited: New insights into the localization, meaning and network organization , 2014, NeuroImage.

[39]  Clay B. Holroyd,et al.  The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. , 2002, Psychological review.

[40]  N. Daw,et al.  Generalization of value in reinforcement learning by humans , 2012, The European journal of neuroscience.

[41]  John R. Anderson,et al.  Learning from experience: Event-related potential correlates of reward processing, neural adaptation, and behavioral choice , 2012, Neuroscience & Biobehavioral Reviews.

[42]  Antonio Rangel,et al.  Decision value computation in DLPFC and VMPFC adjusts to the available decision time , 2012, The European journal of neuroscience.

[43]  J. Gläscher,et al.  Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-related decision making. , 2009, Cerebral cortex.

[44]  J. O'Doherty,et al.  The Role of the Ventromedial Prefrontal Cortex in Abstract State-Based Inference during Decision Making in Humans , 2006, The Journal of Neuroscience.

[45]  L. Deserno,et al.  Model-Based and Model-Free Decisions in Alcohol Dependence , 2014, Neuropsychobiology.

[46]  Antonio Rangel,et al.  Neural computations associated with goal-directed choice , 2010, Current Opinion in Neurobiology.

[47]  R. Dolan,et al.  Ventral striatal prediction error signaling is associated with dopamine synthesis capacity and fluid intelligence , 2013, Human brain mapping.

[48]  Q. Huys,et al.  Failure modes of the will: from goals to habits to compulsions? , 2015, The American journal of psychiatry.

[49]  Daeyeol Lee,et al.  Distributed Coding of Actual and Hypothetical Outcomes in the Orbital and Dorsolateral Prefrontal Cortex , 2011, Neuron.

[50]  Geoffrey Schoenbaum,et al.  Neural Estimates of Imagined Outcomes in the Orbitofrontal Cortex Drive Behavior and Learning , 2013, Neuron.

[51]  G. Schoenbaum,et al.  Transition from ‘model-based’ to ‘model-free’ behavioral control in addiction: Involvement of the orbitofrontal cortex and dorsolateral striatum , 2014, Neuropharmacology.

[52]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[53]  Nathaniel D. Daw,et al.  Trial-by-trial data analysis using computational models , 2011 .

[54]  P. Dayan,et al.  Depression: a decision-theoretic analysis. , 2015, Annual review of neuroscience.

[55]  Guillem R. Esber,et al.  Surprise! Neural correlates of Pearce–Hall and Rescorla–Wagner coexist within the brain , 2012, The European journal of neuroscience.

[56]  Jody Tanabe,et al.  Reduced neural tracking of prediction error in substance-dependent individuals. , 2013, The American journal of psychiatry.

[57]  Peter Dayan,et al.  Decision-Theoretic Psychiatry , 2015 .

[58]  Amir Dezfouli,et al.  Speed/Accuracy Trade-Off between the Habitual and the Goal-Directed Processes , 2011, PLoS Comput. Biol..

[59]  Y. Niv,et al.  The impact of orbitofrontal dysfunction on cocaine addiction , 2012, Nature Neuroscience.

[60]  Markus Ullsperger,et al.  Real and Fictive Outcomes Are Processed Differently but Converge on a Common Adaptive Mechanism , 2013, Neuron.

[61]  Adrian G. Fischer,et al.  Neural mechanisms and temporal dynamics of performance monitoring , 2014, Trends in Cognitive Sciences.

[62]  Karl J. Friston,et al.  Generalisability, Random Effects & Population Inference , 1998, NeuroImage.

[63]  Rita Z. Goldstein,et al.  Impaired Neural Response to Negative Prediction Errors in Cocaine Addiction , 2015, The Journal of Neuroscience.

[64]  P. Dayan,et al.  Goals and Habits in the Brain , 2013, Neuron.