An MEG signature corresponding to an axiomatic model of reward prediction error

Optimal decision-making is guided by evaluating the outcomes of previous decisions. Prediction errors are theoretical teaching signals which integrate two features of an outcome: its inherent value and prior expectation of its occurrence. To uncover the magnetic signature of prediction errors in the human brain we acquired magnetoencephalographic (MEG) data while participants performed a gambling task. Our primary objective was to use formal criteria, based upon an axiomatic model (Caplin and Dean, 2008a), to determine the presence and timing profile of MEG signals that express prediction errors. We report analyses at the sensor level, implemented in SPM8, time locked to outcome onset. We identified, for the first time, a MEG signature of prediction error, which emerged approximately 320 ms after an outcome and expressed as an interaction between outcome valence and probability. This signal followed earlier, separate signals for outcome valence and probability, which emerged approximately 200 ms after an outcome. Strikingly, the time course of the prediction error signal, as well as the early valence signal, resembled the Feedback-Related Negativity (FRN). In simultaneously acquired EEG data we obtained a robust FRN, but the win and loss signals that comprised this difference wave did not comply with the axiomatic model. Our findings motivate an explicit examination of the critical issue of timing embodied in computational models of prediction errors as seen in human electrophysiological data.

[1]  P. Glimcher,et al.  Statistics of midbrain dopamine neuron spike trains in the awake primate. , 2007, Journal of neurophysiology.

[2]  R. Hertwig,et al.  The description–experience gap in risky choice , 2009, Trends in Cognitive Sciences.

[3]  Karl J. Friston,et al.  Statistical parametric mapping for event-related potentials (II): a hierarchical temporal model , 2004, NeuroImage.

[4]  Clay B. Holroyd,et al.  The feedback-related negativity reflects the binary evaluation of good versus bad outcomes , 2006, Biological Psychology.

[5]  A. Sanfey,et al.  Independent Coding of Reward Magnitude and Valence in the Human Brain , 2004, The Journal of Neuroscience.

[6]  Ivan Toni,et al.  Neural dynamics of error processing in medial frontal cortex , 2005, NeuroImage.

[7]  Clay B. Holroyd,et al.  ERP correlates of feedback and reward processing in the presence and absence of response choice. , 2005, Cerebral cortex.

[8]  C. Holroyd,et al.  Which way do I go? Neural activation in response to feedback and spatial processing in a virtual T-maze. , 2009, Cerebral cortex.

[9]  David Cucurell,et al.  Human oscillatory activity associated to reward processing in a gambling task , 2008, Neuropsychologia.

[10]  Karl J. Friston,et al.  Optimized beamforming for simultaneous MEG and intracranial local field potential recordings in deep brain stimulation patients , 2010, NeuroImage.

[11]  Karl J. Friston,et al.  Temporal Difference Models and Reward-Related Learning in the Human Brain , 2003, Neuron.

[12]  P. Dayan,et al.  How Humans Integrate the Prospects of Pain and Reward during Choice , 2009, The Journal of Neuroscience.

[13]  W. Schultz,et al.  Behavioral and Brain Functions , 2005 .

[14]  H. Begleiter,et al.  Brain signatures of monetary loss and gain: Outcome-related potentials in a single outcome gambling task , 2009, Behavioural Brain Research.

[15]  C. Ranganath,et al.  Behavioral and neural predictors of upcoming decisions , 2005, Cognitive, affective & behavioral neuroscience.

[16]  Adrian R. Willoughby,et al.  The Medial Frontal Cortex and the Rapid Processing of Monetary Gains and Losses , 2002, Science.

[17]  Clay B. Holroyd,et al.  Brain potentials associated with expected and unexpected good and bad outcomes. , 2005, Psychophysiology.

[18]  P. Tobler,et al.  Functional imaging of the human dopaminergic midbrain , 2009, Trends in Neurosciences.

[19]  Clay B. Holroyd,et al.  The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. , 2002, Psychological review.

[20]  Shozo Tobimatsu,et al.  Paradoxical lateralization of parasagittal spikes revealed by back averaging of EEG and MEG in a case with epilepsia partialis continua , 2002, Journal of the Neurological Sciences.

[21]  R. Rescorla A theory of pavlovian conditioning: The effectiveness of reinforcement and non-reinforcement , 1972 .

[22]  P. Glimcher,et al.  Testing the Reward Prediction Error Hypothesis with an Axiomatic Model , 2010, The Journal of Neuroscience.

[23]  C. Braun,et al.  Event-Related Brain Potentials Following Incorrect Feedback in a Time-Estimation Task: Evidence for a Generic Neural System for Error Detection , 1997, Journal of Cognitive Neuroscience.

[24]  Jeff T. Larsen,et al.  The good, the bad and the neutral: Electrophysiological responses to feedback stimuli , 2006, Brain Research.

[25]  Margot J. Taylor,et al.  Spatiotemporal analysis of feedback processing during a card sorting task using spatially filtered MEG , 2006, Neuroscience Letters.

[26]  J. Ebersole,et al.  Intracranial EEG Substrates of Scalp EEG Interictal Spikes , 2005, Epilepsia.

[27]  Karl J. Friston,et al.  Statistical parametric mapping for event-related potentials: I. Generic considerations , 2004, NeuroImage.

[28]  E. John,et al.  Evoked-Potential Correlates of Stimulus Uncertainty , 1965, Science.

[29]  Hauke R. Heekeren,et al.  Temporal dynamics of prediction error processing during reward-based decision making , 2010, NeuroImage.

[30]  B De Martino,et al.  The Neurobiology of Reference-Dependent Value Computation , 2009, NeuroImage.

[31]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[32]  Clay B. Holroyd,et al.  Implementation of error-processing in the human anterior cingulate cortex: a source analysis of the magnetic equivalent of the error-related negativity , 2003, Biological Psychology.

[33]  Karl J. Friston,et al.  A Multivariate Analysis of Evoked Responses in EEG and MEG Data , 1996, NeuroImage.

[34]  R. Baker,et al.  When is an error not a prediction error? An electrophysiological investigation , 2009, Cognitive, affective & behavioral neuroscience.

[35]  Tomifusa Kuboki,et al.  Error-related negativity reflects detection of negative reward prediction error , 2004, Neuroreport.

[36]  Jeff T. Larsen,et al.  Context dependence of the event-related brain potential associated with reward and punishment. , 2004, Psychophysiology.

[37]  A. Dickinson,et al.  Neuronal coding of prediction errors. , 2000, Annual review of neuroscience.

[38]  G. Elliott Wimmer,et al.  Neural Antecedents of the Endowment Effect , 2008, Neuron.

[39]  P. Montague,et al.  Efficient statistics, common currencies and the problem of reward-harvesting , 2007, Trends in Cognitive Sciences.

[40]  Clay B. Holroyd,et al.  It's worse than you thought: the feedback negativity and violations of reward prediction in gambling tasks. , 2007, Psychophysiology.

[41]  P. Redgrave,et al.  Is the short-latency dopamine response too short to signal reward error? , 1999, Trends in Neurosciences.

[42]  Nico Bunzeck,et al.  Reward Motivation Accelerates the Onset of Neural Novelty Signals in Humans to 85 Milliseconds , 2009, Current Biology.

[43]  Karl J. Friston The free-energy principle: a unified brain theory? , 2010, Nature Reviews Neuroscience.

[44]  Francisco Barceló,et al.  Spatiotemporal brain dynamics during preparatory set shifting: MEG evidence , 2004, NeuroImage.

[45]  Clay B. Holroyd,et al.  Reinforcement-related brain potentials from medial frontal cortex: origins and functional significance , 2004, Neuroscience & Biobehavioral Reviews.

[46]  S. Haber,et al.  The Reward Circuit: Linking Primate Anatomy and Human Imaging , 2010, Neuropsychopharmacology.

[47]  Matthew C. Keller,et al.  Increased sensitivity in neuroimaging analyses using robust regression , 2005, NeuroImage.

[48]  B. Balleine,et al.  Motivational control of goal-directed action , 1994 .

[49]  Andrew Caplin,et al.  Dopamine, Reward Prediction Error, and Economics , 2008 .

[50]  William J. Gehring,et al.  Medial prefrontal cortex and error potentials , 2002 .

[51]  Clay B. Holroyd,et al.  Errors in reward prediction are re£ected in the event-related brain potential , 2003 .

[52]  S. Debener,et al.  Trial-by-Trial Fluctuations in the Event-Related Electroencephalogram Reflect Dynamic Changes in the Degree of Surprise , 2008, The Journal of Neuroscience.

[53]  Michael X. Cohen,et al.  Reward expectation modulates feedback-related negativity and EEG spectra , 2007, NeuroImage.

[54]  G Nolte,et al.  Partial signal space projection for artefact removal in MEG measurements: a theoretical analysis. , 2001, Physics in medicine and biology.

[55]  Atsushi Sato,et al.  Effects of value and reward magnitude on feedback negativity and P300 , 2005, Neuroreport.

[56]  R. Dolan,et al.  Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans , 2006, Nature.

[57]  D. Kumaran,et al.  Frames, Biases, and Rational Decision-Making in the Human Brain , 2006, Science.

[58]  Clay B. Holroyd,et al.  The feedback correct-related positivity: sensitivity of the event-related brain potential to unexpected positive feedback. , 2008, Psychophysiology.

[59]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[60]  Guillem R. Esber,et al.  All that glitters ... dissociating attention and outcome expectancy from prediction errors signals. , 2010, Journal of neurophysiology.

[61]  E. Donchin Presidential address, 1980. Surprise!...Surprise? , 1981, Psychophysiology.

[62]  Society for Psychophysiological Research 39th annual meeting. Granada, Spain, October 6-10, 1999. Abstracts. , 1999, Psychophysiology.

[63]  Clay B. Holroyd,et al.  Reward prediction error signals associated with a modified time estimation task. , 2007, Psychophysiology.

[64]  D. Kumaran,et al.  The Neurobiology of Reference-Dependent Value Computation , 2009, NeuroImage.

[65]  Monica Borges,et al.  Society for Psychophysiological Research , 2005 .

[66]  Peter Dayan,et al.  Temporal difference models describe higher-order learning in humans , 2004, Nature.

[67]  F. Xavier Castellanos,et al.  Shifting-Related Brain Magnetic Activity in Attention-Deficit/Hyperactivity Disorder , 2006, Biological Psychiatry.

[68]  Andrew Caplin,et al.  Axiomatic Methods, Dopamine and Reward Prediction Error This Review Comes from a Themed Issue on Cognitive Neuroscience Edited Advantages of the Axiomatic Approach , 2022 .