Positive reward prediction errors strengthen incidental memory encoding

The dopamine system is thought to provide a reward prediction error signal that facilitates reinforcement learning and reward-based choice in corticostriatal circuits. While it is believed that similar prediction error signals are also provided to temporal lobe memory systems, the impact of such signals on episodic memory encoding has not been fully characterized. Here we develop an incidental memory paradigm that allows us to 1) estimate the influence of reward prediction errors on the formation of episodic memories, 2) dissociate this influence from other factors such as surprise and uncertainty, 3) test the degree to which this influence depends on temporal correspondence between prediction error and memoranda presentation, and 4) determine the extent to which this influence is consolidation-dependent. We find that when choosing to gamble for potential rewards during a primary decision making task, people encode incidental memoranda more strongly even though they are not aware that their memory will be subsequently probed. Moreover, this strengthened encoding scales with the reward prediction error, and not overall reward, experienced selectively at the time of memoranda presentation (and not before or after). Finally, this strengthened encoding is identifiable within a few minutes and is not substantially enhanced after twenty-four hours, indicating that it is not consolidation-dependent. These results suggest a computationally and temporally specific role for putative dopaminergic reward prediction error signaling in memory formation.

[1]  Malcolm W. Brown,et al.  Different Contributions of the Hippocampus and Perirhinal Cortex to Recognition Memory , 1999, The Journal of Neuroscience.

[2]  S. Floresco,et al.  Dopaminergic regulation of limbic-striatal interplay. , 2007, Journal of psychiatry & neuroscience : JPN.

[3]  Robert C. Wilson,et al.  Rational regulation of learning dynamics by pupil–linked arousal systems , 2012, Nature Neuroscience.

[4]  D. Shohamy,et al.  Dopamine and adaptive memory , 2010, Trends in Cognitive Sciences.

[5]  H. Eichenbaum,et al.  The medial temporal lobe and recognition memory. , 2007, Annual review of neuroscience.

[6]  R. Dolan,et al.  Basal Ganglia Activity Mirrors a Benefit of Action and Reward on Long-Lasting Event Memory , 2015, Cerebral cortex.

[7]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[8]  Angela J. Yu,et al.  Uncertainty, Neuromodulation, and Attention , 2005, Neuron.

[9]  S. Floresco,et al.  Overriding Phasic Dopamine Signals Redirects Action Selection during Risk/Reward Decision Making , 2014, Neuron.

[10]  L. Squire Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans. , 1992, Psychological review.

[11]  Talia N. Lerner,et al.  Nucleus accumbens D2R cells signal prior outcomes and control risky decision-making , 2016, Nature.

[12]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[13]  Anne G E Collins,et al.  Opponent actor learning (OpAL): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive. , 2014, Psychological review.

[14]  J. Glowinski,et al.  Hippocampo‐prefrontal cortex pathway: Anatomical and electrophysiological characteristics , 2000, Hippocampus.

[15]  P. Glimcher,et al.  Testing the Reward Prediction Error Hypothesis with an Axiomatic Model , 2010, The Journal of Neuroscience.

[16]  W. Einhäuser,et al.  Pupil Dilation Signals Surprise: Evidence for Noradrenaline’s Role in Decision Making , 2011, Front. Neurosci..

[17]  P. Glimcher,et al.  Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal , 2005, Neuron.

[18]  Jessica K. Stanek,et al.  Expected reward value and reward uncertainty have temporally dissociable effects on memory formation , 2019, J. Cogn. Neurosci..

[19]  Nicole M. Long,et al.  Hippocampal Mismatch Signals Are Modulated by the Strength of Neural Predictions and Their Similarity to Outcomes , 2016, The Journal of Neuroscience.

[20]  Lila Davachi,et al.  Selectivity in Postencoding Connectivity with High-Level Visual Cortex Is Associated with Reward-Motivated Memory , 2017, The Journal of Neuroscience.

[21]  P. Dayan,et al.  Dopaminergic Modulation of Decision Making and Subjective Well-Being , 2015, The Journal of Neuroscience.

[22]  James L. McClelland,et al.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. , 1995, Psychological review.

[23]  R. Morris,et al.  Dopamine and Memory: Modulation of the Persistence of Memory for Novel Hippocampal NMDA Receptor-Dependent Paired Associates , 2010, The Journal of Neuroscience.

[24]  K. Berridge,et al.  What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? , 1998, Brain Research Reviews.

[25]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[26]  Michael J. Frank,et al.  By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism , 2004, Science.

[27]  Robert C. Wilson,et al.  An Approximately Bayesian Delta-Rule Model Explains the Dynamics of Belief Updating in a Changing Environment , 2010, The Journal of Neuroscience.

[28]  R. O’Reilly,et al.  Conjunctive representations in learning and memory: principles of cortical and hippocampal function. , 2001, Psychological review.

[29]  N. Lemon,et al.  Dopamine D1/D5 Receptors Gate the Acquisition of Novel Information through Hippocampal Long-Term Potentiation and Long-Term Depression , 2006, The Journal of Neuroscience.

[30]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[31]  S. Siegelbaum,et al.  Midbrain dopamine neurons bidirectionally regulate CA3-CA1 synaptic drive , 2015, Nature Neuroscience.

[32]  Suzanne N. Haber,et al.  Circuit-Based Corticostriatal Homologies Between Rat and Primate , 2016, Biological Psychiatry.

[33]  Alison Adcock,et al.  Enriched encoding: reward motivation organizes cortical networks for hippocampal detection of unexpected events. , 2014, Cerebral cortex.

[34]  H. Heinze,et al.  Reward-Related fMRI Activation of Dopaminergic Midbrain Is Associated with Enhanced Hippocampus- Dependent Long-Term Memory Formation , 2005, Neuron.

[35]  Y. Niv,et al.  Dissociable effects of surprising rewards on learning and memory , 2017, bioRxiv.

[36]  R. Henson,et al.  Does prediction error drive one-shot declarative learning? , 2017, Journal of memory and language.

[37]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[38]  S. Sara,et al.  Network reset: a simplified overarching theory of locus coeruleus noradrenaline function , 2005, Trends in Neurosciences.

[39]  W. Schultz,et al.  Discrete Coding of Reward Probability and Uncertainty by Dopamine Neurons , 2003, Science.

[40]  Anne E Carpenter,et al.  Neuron-type specific signals for reward and punishment in the ventral tegmental area , 2011, Nature.

[41]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[42]  H. Eichenbaum Prefrontal–hippocampal interactions in episodic memory , 2017, Nature Reviews Neuroscience.

[43]  Josiah R. Boivin,et al.  A Causal Link Between Prediction Errors, Dopamine Neurons and Learning , 2013, Nature Neuroscience.

[44]  Jessica K. Stanek,et al.  Expected reward value and reward uncertainty have temporally dissociable effects on memory formation , 2018, bioRxiv.

[45]  Michael J. Frank,et al.  Hippocampus, cortex, and basal ganglia: Insights from computational models of complementary learning systems , 2004, Neurobiology of Learning and Memory.

[46]  Geoffrey Schoenbaum,et al.  Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework , 2016, eLife.

[47]  V. Bolshakov,et al.  Emotional enhancement of memory: how norepinephrine enables synaptic plasticity , 2010, Molecular Brain.

[48]  K. Deisseroth,et al.  Parvalbumin neurons and gamma rhythms enhance cortical circuit performance , 2009, Nature.

[49]  U. Frey,et al.  Synaptic tagging: implications for late maintenance of hippocampal long-term potentiation , 1998, Trends in Neurosciences.

[50]  L. Nadel,et al.  Memory consolidation, retrograde amnesia and the hippocampal complex , 1997, Current Opinion in Neurobiology.

[51]  M. Walton,et al.  Dissociable cost and benefit encoding of future rewards by mesolimbic dopamine , 2009, Nature Neuroscience.

[52]  Joseph T. McGuire,et al.  Functionally Dissociable Influences on Learning Rate in a Dynamic Environment , 2014, Neuron.

[53]  Matthew R Nassar,et al.  Age differences in learning emerge from an insufficient representation of uncertainty in older adults , 2016, Nature Communications.

[54]  Erin Kendall Braun,et al.  Episodic Memory Encoding Interferes with Reward Learning and Decreases Striatal Prediction Errors , 2014, The Journal of Neuroscience.

[55]  J. Lisman,et al.  The Hippocampal-VTA Loop: Controlling the Entry of Information into Long-Term Memory , 2005, Neuron.

[56]  D. Dupret,et al.  Dopaminergic neurons promote hippocampal reactivation and spatial memory persistence , 2014, Nature Neuroscience.

[57]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.