tHe Cognitive neuroSCienCe of Motivation and learning

Recent advances in the cognitive neuroscience of motivation and learning have demonstrated a critical role for midbrain dopamine and its targets in reward prediction. Converging evidence suggests that midbrain dopamine neurons signal a reward prediction error, allowing an organism to predict, and to act to increase, the probability of reward in the future. This view has been highly successful in accounting for a wide range of reinforcement learning phenomena in animals and humans. However, while current theories of midbrain dopamine provide a good account of behavior known as habitual or stimulus-response learning, we review evidence suggesting that other neural and cognitive processes are involved in motivated, goal-directed behavior. We discuss how this distinction resembles the classic distinction in the cognitive neuroscience of memory between nondeclarative and declarative memory systems, and discuss common themes between mnemonic and motivational functions. Finally, we present data demonstrating links between mnemonic processes and reinforcement learning.

[1]  G. Grice,et al.  Mediated stimulus equivalence and distinctiveness in human conditioning. , 1958, Journal of experimental psychology.

[2]  G. Grice,et al.  Effect of concurrent responses on the evocation and generalization of the conditioned eyeblink. , 1960, Journal of experimental psychology.

[3]  R. Rescorla,et al.  A theory of Pavlovian conditioning : Variations in the effectiveness of reinforcement and nonreinforcement , 1972 .

[4]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[5]  J. Winn,et al.  Brain , 1878, The Lancet.

[6]  L. Squire,et al.  Preserved learning and retention of pattern-analyzing skill in amnesia: dissociation of knowing how and knowing that. , 1980, Science.

[7]  S. Lea,et al.  Contemporary Animal Learning Theory, Anthony Dickinson. Cambridge University Press, Cambridge (1981), xii, +177 pp. £12.50 hardback, £3.95 paperback , 1981 .

[8]  L. Squire,et al.  Retrograde amnesia and remote memory impairment , 1981, Neuropsychologia.

[9]  H. Spinnler.  The prefrontal cortex, Anatomy, physiology, and neuropsychology of the frontal lobe, J.M. Fuster. Raven Press, New York (1980), IX-222 pages , 1981 .

[10]  Christopher D. Adams.  Variations in the Sensitivity of Instrumental Responding to Reinforcer Devaluation , 1982 .

[11]  P. Goldman-Rakic,et al.  Dual pathways connecting the dorsolateral prefrontal cortex with the hippocampal formation and parahippocampal cortex in the rhesus monkey , 1984, Neuroscience.

[12]  M. M. Patterson,et al.  Fimbrial lesions and sensory preconditioning. , 1984, Behavioral neuroscience.

[13]  G. E. Alexander,et al.  Parallel organization of functionally segregated circuits linking basal ganglia and cortex. , 1986, Annual review of neuroscience.

[14]  L. Squire.  The organization and neural substrates of human memory. , 1987, International journal of neurology.

[15]  M. Packard,et al.  Differential effects of fornix and caudate nucleus lesions on two radial maze tasks: evidence for multiple memory systems , 1989, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[16]  J. Penney,et al.  The functional anatomy of basal ganglia disorders , 1989, Trends in Neurosciences.

[17]  M. Delong,et al.  Primate models of movement disorders of basal ganglia origin , 1990, Trends in Neurosciences.

[18]  H. Eichenbaum,et al.  Hippocampal representation in place learning , 1990, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[19]  John A. Bargh,et al.  Auto-motives: Preconscious determinants of social interaction. , 1990 .

[20]  A. Graybiel,et al.  The nigrostriatal system in Parkinson's disease. , 1990, Advances in neurology.

[21]  L. Squire.  Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans. , 1992, Psychological review.

[22]  G. Hall,et al.  Acquired equivalence between cues trained with a common antecedent. , 1993, Journal of experimental psychology. Animal behavior processes.

[23]  M. Gluck,et al.  Hippocampal mediation of stimulus representation: A computational theory , 1993, Hippocampus.

[24]  Joel L. Davis,et al.  A Model of How the Basal Ganglia Generate and Use Neural Signals That Predict Reinforcement , 1994 .

[25]  J. Hodges.  Memory, Amnesia and the Hippocampal System , 1995 .

[26]  R. Boakes,et al.  Motivational control after extended instrumental training , 1995 .

[27]  A. Barto.  Adaptive Critics and the Basal Ganglia , 1995 .

[28]  T. Robbins,et al.  Refining the Taxonomy of Memory , 1996, Science.

[29]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[30]  Jennifer A. Mangels,et al.  A Neostriatal Habit Learning System in Humans , 1996, Science.

[31]  H. Eichenbaum,et al.  Conservation of hippocampal memory function in rats and humans , 1996, Nature.

[32]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[33]  N. White.  Mnemonic functions of the basal ganglia , 1997, Current Opinion in Neurobiology.

[34]  A. Owen.  Cognitive planning in humans: Neuropsychological, neuroanatomical and neuropharmacological perspectives , 1997, Progress in Neurobiology.

[35]  A. Damasio,et al.  Deciding Advantageously Before Knowing the Advantageous Strategy , 1997, Science.

[36]  H. Eichenbaum,et al.  The hippocampus and memory for orderly stimulus relations. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[37]  J. Desmond,et al.  Making memories: brain activity that predicts how well visual experience will be remembered. , 1998, Science.

[38]  J. Gabrieli.  Cognitive neuroscience of human memory. , 1998, Annual review of psychology.

[39]  A. Dale,et al.  Building memories: remembering and forgetting of verbal experiences as predicted by brain activity. , 1998, Science.

[40]  B. Balleine,et al.  Goal-directed instrumental action: contingency and incentive learning and their cortical substrates , 1998, Neuropharmacology.

[41]  W. Schultz,et al.  The phasic reward signal of primate dopamine neurons. , 1998, Advances in pharmacology.

[42]  M. Chun,et al.  Memory deficits for implicit contextual information in amnesic subjects with hippocampal damage , 1999, Nature Neuroscience.

[43]  D. Schacter,et al.  Medial temporal lobe activations in fMRI and PET studies of episodic encoding and retrieval , 1999, Hippocampus.

[44]  P. Redgrave,et al.  Is the short-latency dopamine response too short to signal reward error? , 1999, Trends in Neurosciences.

[45]  D. Schacter,et al.  When encoding yields remembering: insights from event-related neuroimaging. , 1999, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[46]  A. Dickinson,et al.  Episodic memory: what can animals remember about their past? , 1999, Trends in Cognitive Sciences.

[47]  R. Spinks.  The Prefrontal Cortex: Anatomy, Physiology, and Neuropsychology of the Frontal Lobe, 3rd ed. , 2000 .

[48]  B. Balleine,et al.  The Role of the Hippocampus in Instrumental Conditioning , 2000, The Journal of Neuroscience.

[49]  C. Stern,et al.  Prefrontal–Temporal Circuitry for Episodic Encoding and Subsequent Memory , 2000, The Journal of Neuroscience.

[50]  J. Horvitz.  Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events , 2000, Neuroscience.

[51]  L. Nystrom,et al.  Tracking the hemodynamic responses to reward and punishment in the striatum. , 2000, Journal of neurophysiology.

[52]  L. Davachi,et al.  Cognitive neuroscience: Forgetting of things past , 2001, Current Biology.

[53]  N. Cohen.  From Conditioning to Conscious Recollection Memory Systems of the Brain. Oxford Psychology Series, Volume 35. , 2001 .

[54]  M. Gluck,et al.  Interactive memory systems in the human brain , 2001, Nature.

[55]  R N Henson,et al.  Depth of processing effects on neural correlates of memory encoding: relationship between findings from across- and within-task comparisons. , 2001, Brain : a journal of neurology.

[56]  E. Walther.  Guilty by mere association: evaluative conditioning and the spreading attitude effect. , 2002, Journal of personality and social psychology.

[57]  P. Dayan,et al.  Reward, Motivation, and Reinforcement Learning , 2002, Neuron.

[58]  T. Robbins,et al.  Decision-making processes following damage to the prefrontal cortex. , 2002, Brain : a journal of neurology.

[59]  T. Robbins,et al.  Nucleus accumbens dopamine depletion impairs both acquisition and performance of appetitive Pavlovian approach behaviour: implications for mesoaccumbens dopamine function , 2002, Behavioural Brain Research.

[60]  B. Everitt,et al.  Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex , 2002, Neuroscience & Biobehavioral Reviews.

[61]  J. Horvitz.  Dopamine gating of glutamatergic sensorimotor and incentive motivational input signals to the striatum , 2002, Behavioural Brain Research.

[62]  K. Paller,et al.  Observing the transformation of experience into memory , 2002, Trends in Cognitive Sciences.

[63]  Peter Dayan,et al.  Dopamine: generalization and bonuses , 2002, Neural Networks.

[64]  B. Balleine,et al.  The Role of Learning in the Operation of Motivational Systems , 2002 .

[65]  R. Poldrack,et al.  Competition among multiple memory systems: converging evidence from animal and human brain studies , 2003, Neuropsychologia.

[66]  Samuel M. McClure,et al.  A computational substrate for incentive salience , 2003, Trends in Neurosciences.

[67]  T. Robbins,et al.  l-Dopa medication remediates cognitive inflexibility, but increases impulsivity in patients with Parkinson’s disease , 2003, Neuropsychologia.

[68]  Samuel M. McClure,et al.  Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum , 2003, Neuron.

[69]  Anthony D Wagner,et al.  Memory orientation and success: separable neurocognitive components underlying episodic recognition , 2003, Neuropsychologia.

[70]  Karl J. Friston,et al.  Temporal Difference Models and Reward-Related Learning in the Human Brain , 2003, Neuron.

[71]  B. Balleine,et al.  The Effect of Lesions of the Basolateral Amygdala on Instrumental Conditioning , 2003, The Journal of Neuroscience.

[72]  B. Balleine,et al.  The role of prelimbic cortex in instrumental conditioning , 2003, Behavioural Brain Research.

[73]  W. Schultz,et al.  Discrete Coding of Reward Probability and Uncertainty by Dopamine Neurons , 2003, Science.

[74]  H. Spiers,et al.  Prefrontal and medial temporal lobe interactions in long-term memory , 2003, Nature Reviews Neuroscience.

[75]  M. Gluck,et al.  Dissociating Hippocampal versus Basal Ganglia Contributions to Learning and Transfer , 2003, Journal of Cognitive Neuroscience.

[76]  S. Haber.  The primate basal ganglia: parallel and integrative networks , 2003, Journal of Chemical Neuroanatomy.

[77]  Tatsuo K Sato,et al.  Correlated Coding of Motivation and Outcome of Decision by Dopamine Neurons , 2003, The Journal of Neuroscience.

[78]  Matthew D. Lieberman,et al.  Reflexive and reflective judgment processes: A social cognitive neuroscience approach. , 2003 .

[79]  S. Killcross,et al.  Coordination of actions and habits in the medial prefrontal cortex of rats. , 2003, Cerebral cortex.

[80]  M. Gluck,et al.  Human midbrain sensitivity to cognitive feedback and uncertainty during classification learning. , 2004, Journal of neurophysiology.

[81]  Saori C. Tanaka,et al.  Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops , 2004, Nature Neuroscience.

[82]  D. Amaral,et al.  Functional Neuroanatomy of the Medial Temporal Lobe Memory System , 2004, Cortex.

[83]  Samuel M. McClure,et al.  Neural Correlates of Behavioral Preference for Culturally Familiar Drinks , 2004, Neuron.

[84]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[85]  Samuel M. McClure,et al.  Separate Neural Systems Value Immediate and Delayed Monetary Rewards , 2004, Science.

[86]  M. Gluck,et al.  Cortico-striatal contributions to feedback-based learning: converging data from neuroimaging and neuropsychology. , 2004, Brain : a journal of neurology.

[87]  Neal J. Cohen,et al.  Processing and short-term retention of relational information in amnesia , 2004, Neuropsychologia.

[88]  Michael J. Frank,et al.  By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism , 2004, Science.

[89]  E. Murray,et al.  Bilateral Orbital Prefrontal Cortex Lesions in Rhesus Monkeys Disrupt Choices Guided by Both Reward Value and Reward Contingency , 2004, The Journal of Neuroscience.

[90]  B. Balleine,et al.  Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning , 2004, The European journal of neuroscience.

[91]  K. Doya,et al.  A Neural Correlate of Reward-Based Behavioral Learning in Caudate Nucleus: A Functional Magnetic Resonance Imaging Study of a Stochastic Decision Task , 2004, The Journal of Neuroscience.

[92]  L. Davachi,et al.  Behavioral/systems/cognitive Functional–neuroanatomic Correlates of Recollection: Implications for Models of Recognition Memory , 2022 .

[93]  D. Amaral,et al.  Entorhinal Cortex Lesions Disrupt the Relational Organization of Memory in Monkeys , 2004, The Journal of Neuroscience.

[94]  M. Delgado,et al.  Modulation of Caudate Activity by Action Contingency , 2004, Neuron.

[95]  M. Gluck,et al.  Role of the basal ganglia in category learning: how do patients with Parkinson's disease learn? , 2004, Behavioral neuroscience.

[96]  J. O'Doherty,et al.  Reward representations and reward-related learning in the human brain: insights from neuroimaging , 2004, Current Opinion in Neurobiology.

[97]  F. Strack,et al.  Reflective and Impulsive Determinants of Social Behavior , 2004, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[98]  E. Vaadia,et al.  Coincident but Distinct Messages of Midbrain Dopamine and Striatal Tonically Active Neurons , 2004, Neuron.

[99]  H. Heinze,et al.  Reward-Related fMRI Activation of Dopaminergic Midbrain Is Associated with Enhanced Hippocampus- Dependent Long-Term Memory Formation , 2005, Neuron.

[100]  S. Quartz,et al.  Getting to Know You: Reputation and Trust in a Two-Person Economic Exchange , 2005, Science.

[101]  M. Delgado,et al.  Perceptions of moral character modulate the neural systems of reward during the trust game , 2005, Nature Neuroscience.

[102]  Geoffrey Hall,et al.  Acquired distinctiveness and equivalence in human discrimination learning: Evidence for an attentional process , 2005, Psychonomic bulletin & review.

[103]  M. Gluck,et al.  The role of dopamine in cognitive sequence learning: evidence from Parkinson’s disease , 2005, Behavioural Brain Research.

[104]  A. Faure,et al.  Lesion to the Nigrostriatal Dopamine System Disrupts Stimulus-Response Habit Formation , 2005, The Journal of Neuroscience.

[105]  T. Robbins,et al.  Neural systems of reinforcement for drug addiction: from actions to habits to compulsion , 2005, Nature Neuroscience.

[106]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[107]  Michael J. Frank,et al.  Dynamic Dopamine Modulation in the Basal Ganglia: A Neurocomputational Account of Cognitive Deficits in Medicated and Nonmedicated Parkinsonism , 2005, Journal of Cognitive Neuroscience.

[108]  Andy C. H. Lee,et al.  Behavioral / Systems / Cognitive Functional Specialization in the Human Medial Temporal Lobe , 2005 .

[109]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[110]  P. Glimcher,et al.  Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal , 2005, Neuron.

[111]  B. Balleine,et al.  The role of the dorsomedial striatum in instrumental conditioning , 2005, The European journal of neuroscience.

[112]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[113]  P. Dayan,et al.  Dopamine, uncertainty and TD learning , 2005, Behavioral and Brain Functions.

[114]  Russell A. Epstein,et al.  Perceptual deficits in amnesia: challenging the medial temporal lobe ‘mnemonic’ view , 2005, Neuropsychologia.

[115]  R. Palmiter,et al.  Distinguishing whether dopamine regulates liking, wanting, and/or learning about rewards. , 2005, Behavioral neuroscience.

[116]  M. Kawato,et al.  Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. , 2006, Journal of neurophysiology.

[117]  D. Schacter,et al.  Rapid response learning in amnesia: Delineating associative learning components in repetition priming , 2006, Neuropsychologia.

[118]  S. Hyman,et al.  Neural mechanisms of addiction: the role of reward-related learning and memory. , 2006, Annual review of neuroscience.

[119]  K. Berridge.  The debate over dopamine’s role in reward: the case for incentive salience , 2007, Psychopharmacology.

[120]  P. Redgrave,et al.  The short-latency dopamine signal: a role in discovering novel actions? , 2006, Nature Reviews Neuroscience.

[121]  Steven E. Prince,et al.  The Medial Temporal Lobe Distinguishes Old from New Independently of Consciousness , 2006, The Journal of Neuroscience.

[122]  M. Gluck,et al.  l-dopa impairs learning, but spares generalization, in Parkinson's disease , 2006, Neuropsychologia.

[123]  M. Shapiro,et al.  Episodic memory—From brain to mind , 2006, Hippocampus.

[124]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[125]  C. Ranganath,et al.  Dorsolateral Prefrontal Cortex Promotes Long-Term Memory Formation through Its Role in Working Memory Organization , 2006, The Journal of Neuroscience.

[126]  R. Dolan,et al.  Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans , 2006, Nature.

[127]  Brian Knutson,et al.  Reward-Motivated Learning: Mesolimbic Activation Precedes Memory Formation , 2006, Neuron.

[128]  P. Dayan,et al.  Opinion TRENDS in Cognitive Sciences Vol.10 No.8 Full text provided by www.sciencedirect.com A normative perspective on motivation , 2022 .

[129]  J. O'Doherty,et al.  The Role of the Ventromedial Prefrontal Cortex in Abstract State-Based Inference during Decision Making in Humans , 2006, The Journal of Neuroscience.

[130]  J. Salamone.  Functions of mesolimbic dopamine: changing concepts and shifting paradigms , 2007, Psychopharmacology.

[131]  Russell A Poldrack,et al.  Modulation of competing memory systems by distraction. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[132]  E. Vaadia,et al.  Midbrain dopamine neurons encode decisions for future action , 2006, Nature Neuroscience.

[133]  David S. Touretzky,et al.  Representation and Timing in Theories of the Dopamine System , 2006, Neural Computation.

[134]  P. Dayan,et al.  Tonic dopamine: opportunity costs and the control of response vigor , 2007, Psychopharmacology.

[135]  T. Robbins,et al.  A role for mesencephalic dopamine in activation: commentary on Berridge (2006) , 2007, Psychopharmacology.

[136]  K. Szpunar,et al.  Neural substrates of envisioning the future , 2007, Proceedings of the National Academy of Sciences.

[137]  D. Hassabis,et al.  Patients with hippocampal amnesia cannot imagine new experiences , 2007, Proceedings of the National Academy of Sciences.

[138]  M. Roesch,et al.  Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards , 2007, Nature Neuroscience.

[139]  Vivian V. Valentin,et al.  Determining the Neural Substrates of Goal-Directed Learning in the Human Brain , 2007, The Journal of Neuroscience.

[140]  Daniel L. Schacter,et al.  Constructive memory: The ghosts of past and future , 2007, Nature.

[141]  R. Buckner,et al.  Opinion TRENDS in Cognitive Sciences Vol.11 No.2 Self-projection and the brain , 2022 .

[142]  Alana T. Wong,et al.  Remembering the past and imagining the future: Common and distinct neural substrates during event construction and elaboration , 2007, Neuropsychologia.

[143]  D. Schacter,et al.  The cognitive neuroscience of constructive memory: remembering the past and imagining the future , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[144]  J. Gabrieli,et al.  Dissociation between explicit memory and configural memory in the human medial temporal lobe. , 2008, Cerebral cortex.

[145]  R. K. Simpson.  Nature Neuroscience , 2022 .