Positive reward prediction errors during decision making strengthen memory encoding

Dopamine is thought to provide reward prediction error signals to temporal lobe memory systems, but the role of these signals in episodic memory has not been fully characterized. Here we developed an incidental memory paradigm to (i) estimate the influence of reward prediction errors on the formation of episodic memories, (ii) dissociate this influence from surprise and uncertainty, (iii) characterize the role of temporal correspondence between prediction error and memoranda presentation and (iv) determine the extent to which this influence is dependent on memory consolidation. We found that people encoded incidental memoranda more strongly when they gambled for potential rewards. Moreover, the degree to which gambling strengthened encoding scaled with the reward prediction error experienced when memoranda were presented (and not before or after). This encoding enhancement was detectable within minutes and did not differ substantially after 24 h, indicating that it is not dependent on memory consolidation. These results suggest a computationally and temporally specific role for reward prediction error signalling in memory formation.Jang and colleagues show that positive reward prediction errors elicited during incidental encoding enhance the formation of episodic memories.

[1]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[2]  Malcolm W. Brown,et al.  Different Contributions of the Hippocampus and Perirhinal Cortex to Recognition Memory , 1999, The Journal of Neuroscience.

[3]  Anne E Carpenter,et al.  Neuron-type specific signals for reward and punishment in the ventral tegmental area , 2011, Nature.

[4]  James L. McClelland,et al.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. , 1995, Psychological review.

[5]  Talia N. Lerner,et al.  Nucleus accumbens D2R cells signal prior outcomes and control risky decision-making , 2016, Nature.

[6]  G. Winocur,et al.  Episodic Memory and Beyond: The Hippocampus and Neocortex in Transformation. , 2016, Annual review of psychology.

[7]  R. Morris,et al.  Dopamine and Memory: Modulation of the Persistence of Memory for Novel Hippocampal NMDA Receptor-Dependent Paired Associates , 2010, The Journal of Neuroscience.

[8]  K. Berridge,et al.  What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? , 1998, Brain Research Reviews.

[9]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[10]  N. Lemon,et al.  Dopamine D1/D5 Receptors Gate the Acquisition of Novel Information through Hippocampal Long-Term Potentiation and Long-Term Depression , 2006, The Journal of Neuroscience.

[11]  Lila Davachi,et al.  Selectivity in Postencoding Connectivity with High-Level Visual Cortex Is Associated with Reward-Motivated Memory , 2017, The Journal of Neuroscience.

[12]  Barbara J Knowlton,et al.  Value-based modulation of memory encoding involves strategic engagement of fronto-temporal semantic processing regions , 2014, Cognitive, Affective, & Behavioral Neuroscience.

[13]  H. Eichenbaum Prefrontal–hippocampal interactions in episodic memory , 2017, Nature Reviews Neuroscience.

[14]  K. Doya Modulators of decision making , 2008, Nature Neuroscience.

[15]  Josiah R. Boivin,et al.  A Causal Link Between Prediction Errors, Dopamine Neurons and Learning , 2013, Nature Neuroscience.

[16]  R. Dolan,et al.  Basal Ganglia Activity Mirrors a Benefit of Action and Reward on Long-Lasting Event Memory , 2015, Cerebral cortex.

[17]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[18]  S. Siegelbaum,et al.  Midbrain dopamine neurons bidirectionally regulate CA3-CA1 synaptic drive , 2015, Nature Neuroscience.

[19]  J. Lisman,et al.  The Hippocampal-VTA Loop: Controlling the Entry of Information into Long-Term Memory , 2005, Neuron.

[20]  S. Gershman,et al.  Dopamine reward prediction errors reflect hidden state inference across time , 2017, Nature Neuroscience.

[21]  D. Dupret,et al.  Dopaminergic neurons promote hippocampal reactivation and spatial memory persistence , 2014, Nature Neuroscience.

[22]  S. Floresco,et al.  Dopaminergic regulation of limbic-striatal interplay. , 2007, Journal of psychiatry & neuroscience : JPN.

[23]  Geoffrey Schoenbaum,et al.  Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework , 2016, eLife.

[24]  M. Gluck,et al.  Dopaminergic Drugs Modulate Learning Rates and Perseveration in Parkinson's Patients in a Dynamic Foraging Task , 2009, The Journal of Neuroscience.

[25]  Michael D. Howard,et al.  Complementary Learning Systems , 2014, Cogn. Sci..

[26]  Suzanne N. Haber,et al.  Circuit-Based Corticostriatal Homologies Between Rat and Primate , 2016, Biological Psychiatry.

[27]  V. Bolshakov,et al.  Emotional enhancement of memory: how norepinephrine enables synaptic plasticity , 2010, Molecular Brain.

[28]  Anthony I. Jang,et al.  Human Cortical Neurons in the Anterior Temporal Lobe Reinstate Spiking Activity during Verbal Memory Retrieval , 2017, Current Biology.

[29]  Michael J. Frank,et al.  Hippocampus, cortex, and basal ganglia: Insights from computational models of complementary learning systems , 2004, Neurobiology of Learning and Memory.

[30]  K. Deisseroth,et al.  Parvalbumin neurons and gamma rhythms enhance cortical circuit performance , 2009, Nature.

[31]  U. Frey,et al.  Synaptic tagging: implications for late maintenance of hippocampal long-term potentiation , 1998, Trends in Neurosciences.

[32]  Wolfram Schultz,et al.  Dopamine reward prediction-error signalling: a two-component response , 2016, Nature Reviews Neuroscience.

[33]  Y. Niv,et al.  Dissociable effects of surprising rewards on learning and memory , 2017, bioRxiv.

[34]  L. Squire Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans. , 1992, Psychological review.

[35]  W. Einhäuser,et al.  Pupil Dilation Signals Surprise: Evidence for Noradrenaline’s Role in Decision Making , 2011, Front. Neurosci..

[36]  Jessica K. Stanek,et al.  Expected reward value and reward uncertainty have temporally dissociable effects on memory formation , 2019, J. Cogn. Neurosci..

[37]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[38]  Tom Verguts,et al.  Signed reward prediction errors drive declarative learning , 2018, PloS one.

[39]  Nicole M. Long,et al.  Hippocampal Mismatch Signals Are Modulated by the Strength of Neural Predictions and Their Similarity to Outcomes , 2016, The Journal of Neuroscience.

[40]  J. Glowinski,et al.  Hippocampo‐prefrontal cortex pathway: Anatomical and electrophysiological characteristics , 2000, Hippocampus.

[41]  A. Dagher,et al.  The role of dopamine in risk taking: a specific look at Parkinson’s disease and gambling , 2014, Front. Behav. Neurosci..

[42]  Matthew R Nassar,et al.  Age differences in learning emerge from an insufficient representation of uncertainty in older adults , 2016, Nature Communications.

[43]  Erin Kendall Braun,et al.  Episodic Memory Encoding Interferes with Reward Learning and Decreases Striatal Prediction Errors , 2014, The Journal of Neuroscience.

[44]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[45]  R. Henson,et al.  Does prediction error drive one-shot declarative learning? , 2017, Journal of memory and language.

[46]  Alison Adcock,et al.  Enriched encoding: reward motivation organizes cortical networks for hippocampal detection of unexpected events. , 2014, Cerebral cortex.

[47]  Larry R Squire,et al.  Spatial memory, recognition memory, and the hippocampus. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[48]  T. McHugh,et al.  Noradrenergic modulation of evoked dopamine release and pH shift in the mouse dorsal hippocampus and ventral striatum , 2017, Brain Research.

[49]  H. Heinze,et al.  Reward-Related fMRI Activation of Dopaminergic Midbrain Is Associated with Enhanced Hippocampus- Dependent Long-Term Memory Formation , 2005, Neuron.

[50]  P. Dayan,et al.  Dopaminergic Modulation of Decision Making and Subjective Well-Being , 2015, The Journal of Neuroscience.

[51]  L. Nadel,et al.  Memory consolidation, retrograde amnesia and the hippocampal complex , 1997, Current Opinion in Neurobiology.

[52]  M. Walton,et al.  Dissociable cost and benefit encoding of future rewards by mesolimbic dopamine , 2009, Nature Neuroscience.

[53]  Joseph T. McGuire,et al.  Functionally Dissociable Influences on Learning Rate in a Dynamic Environment , 2014, Neuron.

[54]  R. Clark,et al.  Recognition memory and the medial temporal lobe: a new perspective , 2007, Nature Reviews Neuroscience.

[55]  Michael J. Frank,et al.  By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism , 2004, Science.

[56]  D. Shohamy,et al.  Dopamine and adaptive memory , 2010, Trends in Cognitive Sciences.

[57]  L. Davachi,et al.  The Simple Act of Choosing Influences Declarative Memory , 2015, The Journal of Neuroscience.

[58]  H. Eichenbaum,et al.  The medial temporal lobe and recognition memory. , 2007, Annual review of neuroscience.

[59]  P. Glimcher,et al.  Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal , 2005, Neuron.

[60]  M. Kahana,et al.  Human Substantia Nigra Neurons Encode Unexpected Financial Rewards , 2009, Science.

[61]  Angela J. Yu,et al.  Uncertainty, Neuromodulation, and Attention , 2005, Neuron.

[62]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[63]  L. Davachi,et al.  Time-resolved neural reinstatement and pattern separation during memory decisions in human hippocampus , 2018, Proceedings of the National Academy of Sciences of the United States of America.

[64]  H. Farmer A new perspective. , 1988, The Journal of the Florida Medical Association.

[65]  S. Floresco,et al.  Overriding Phasic Dopamine Signals Redirects Action Selection during Risk/Reward Decision Making , 2014, Neuron.

[66]  Robert C. Wilson,et al.  Rational regulation of learning dynamics by pupil–linked arousal systems , 2012, Nature Neuroscience.

[67]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[68]  S. Sara,et al.  Network reset: a simplified overarching theory of locus coeruleus noradrenaline function , 2005, Trends in Neurosciences.

[69]  W. Schultz,et al.  Discrete Coding of Reward Probability and Uncertainty by Dopamine Neurons , 2003, Science.

[70]  Anne G E Collins,et al.  Opponent actor learning (OpAL): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive. , 2014, Psychological review.

[71]  Vaughn L. Hetrick,et al.  Mesolimbic Dopamine Signals the Value of Work , 2015, Nature Neuroscience.

[72]  Robert C. Wilson,et al.  An Approximately Bayesian Delta-Rule Model Explains the Dynamics of Belief Updating in a Changing Environment , 2010, The Journal of Neuroscience.