Mapping anhedonia onto reinforcement learning: a behavioural meta-analysis

BackgroundDepression is characterised partly by blunted reactions to reward. However, tasks probing this deficiency have not distinguished insensitivity to reward from insensitivity to the prediction errors for reward that determine learning and are putatively reported by the phasic activity of dopamine neurons. We attempted to disentangle these factors with respect to anhedonia in the context of stress, Major Depressive Disorder (MDD), Bipolar Disorder (BPD) and a dopaminergic challenge.MethodsSix behavioural datasets involving 392 experimental sessions were subjected to a model-based, Bayesian meta-analysis. Participants across all six studies performed a probabilistic reward task that used an asymmetric reinforcement schedule to assess reward learning. Healthy controls were tested under baseline conditions, stress or after receiving the dopamine D2 agonist pramipexole. In addition, participants with current or past MDD or BPD were evaluated. Reinforcement learning models isolated the contributions of variation in reward sensitivity and learning rate.ResultsMDD and anhedonia reduced reward sensitivity more than they affected the learning rate, while a low dose of the dopamine D2 agonist pramipexole showed the opposite pattern. Stress led to a pattern consistent with a mixed effect on reward sensitivity and learning rate.ConclusionReward-related learning reflected at least two partially separable contributions. The first related to phasic prediction error signalling, and was preferentially modulated by a low dose of the dopamine agonist pramipexole. The second related directly to reward sensitivity, and was preferentially reduced in MDD and anhedonia. Stress altered both components. Collectively, these findings highlight the contribution of model-based reinforcement learning meta-analysis for dissecting anhedonic behavior.

[1]  THE INSTITUTE OF RADIO ENGINEERS , 1943, Science.

[2]  M. Hamilton A RATING SCALE FOR DEPRESSION , 1960, Journal of neurology, neurosurgery, and psychiatry.

[3]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[4]  R. Rescorla,et al.  A theory of Pavlovian conditioning : Variations in the effectiveness of reinforcement and nonreinforcement , 1972 .

[5]  C. G. Costello Depression: Loss of reinforcers of loss of reinforcer effectiveness? , 1972 .

[6]  W. F. Prokasy,et al.  Classical conditioning II: Current research and theory. , 1972 .

[7]  H. Akiskal,et al.  Depressive Disorders: Toward a Unified Hypothesis , 1973, Science.

[8]  F. Bloom,et al.  The Biochemical Basis of Neuropharmacology , 1976 .

[9]  R. E. Nelson,et al.  Selective recall of positive and negative feedback, self-control behaviors, and depression. , 1977, Journal of abnormal psychology.

[10]  P. H. Blaney Contemporary theories of depression: critique and comparison. , 1977, Journal of abnormal psychology.

[11]  M. Baudry,et al.  In vivo binding of 3H-pimozide in mouse striatum: effects of dopamine agonists and antagonists. , 1977, Life sciences.

[12]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[13]  F. Jones,et al.  The Liver: An Atlas of Scanning Electron Microscopy , 1978 .

[14]  R. C. Young,et al.  A Rating Scale for Mania: Reliability, Validity and Sensitivity , 1978, British Journal of Psychiatry.

[15]  A G Barto,et al.  Toward a modern theory of adaptive networks: expectation and prediction. , 1981, Psychological review.

[16]  C. Sumners,et al.  Behavioural and neurochemical studies on apomorphine-induced hypomotility in mice , 1981, Neuropharmacology.

[17]  J. Clemens,et al.  Degree of Selectivity of Pergolide as an Agonist at Presynaptic Versus Postsynaptic Dopamine Receptors: Implications for Prevention or Treatment of Tardive Dyskinesia , 1982, Journal of clinical psychopharmacology.

[18]  G. Gessa,et al.  Autoreceptors mediate the inhibition of dopamine synthesis by bromocriptine and lisuride in rats. , 1983, European journal of pharmacology.

[19]  Bernard Widrow,et al.  Adaptive switching circuits , 1988 .

[20]  Richard S. Sutton,et al.  Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming , 1990, ML.

[21]  A. Grace Phasic versus tonic dopamine release and the modulation of dopamine system responsivity: A hypothesis for the etiology of schizophrenia , 1991, Neuroscience.

[22]  S. Kapur,et al.  Role of the dopaminergic system in depression , 1992, Biological Psychiatry.

[23]  J. Rabe-Jabłońska,et al.  [Affective disorders in the fourth edition of the classification of mental disorders prepared by the American Psychiatric Association -- diagnostic and statistical manual of mental disorders]. , 1993, Psychiatria polska.

[24]  Jeffrey B. Henriques,et al.  Reward fails to alter response bias in depression. , 1994, Journal of abnormal psychology.

[25]  H. Meltzer,et al.  Effect of antidepressants on striatal and accumbens extracellular dopamine levels. , 1995, European journal of pharmacology.

[26]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[27]  R A McCormick,et al.  Testing a tripartite model: I. Evaluating the convergent and discriminant validity of anxiety and depression symptom scales. , 1995, Journal of abnormal psychology.

[28]  B. Grant,et al.  Comorbidity between DSM-IV drug use disorders and major depression: results of a national survey of adults. , 1995, Journal of substance abuse.

[29]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[30]  M. Piercey,et al.  Inhibition of dopamine neuron firing by pramipexole, a dopamine D3 receptor-preferring agonist: comparison to other dopamine receptor agonists. , 1996, European journal of pharmacology.

[31]  J. Cummings,et al.  The occurrence of depression in Parkinson's disease. A community-based study. , 1996, Archives of neurology.

[32]  Y. Lecrubier,et al.  Measures of anhedonia and hedonic responses to sucrose in depressive and schizophrenic patients in comparison with healthy subjects , 1998, European Psychiatry.

[33]  R. Elliott,et al.  Abnormal neural response to feedback on planning and guessing tasks in patients with unipolar depression , 1998, Psychological Medicine.

[34]  K. Berridge,et al.  What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? , 1998, Brain Research Reviews.

[35]  B. Alsop,et al.  Sensitivity to reward frequency in boys with attention deficit hyperactivity disorder. , 1999, Journal of clinical child psychology.

[36]  M. Hasselmo Neuromodulation: acetylcholine and memory consolidation , 1999, Trends in Cognitive Sciences.

[37]  T. Robbins,et al.  Dissociation in Effects of Lesions of the Nucleus Accumbens Core and Shell on Appetitive Pavlovian Approach Behavior and the Potentiation of Conditioned Reinforcement and Locomotor Activity byd-Amphetamine , 1999, The Journal of Neuroscience.

[38]  A. Chiba,et al.  Cognitive functions of the basal forebrain , 1999, Current Opinion in Neurobiology.

[39]  J. Mirenowicz,et al.  Dissociation of Pavlovian and instrumental incentive learning under dopamine antagonists. , 2000, Behavioral neuroscience.

[40]  P. D’Aquila,et al.  The role of dopamine in the mechanism of action of antidepressant drugs. , 2000, European journal of pharmacology.

[41]  S. Kakade,et al.  Learning and selective attention , 2000, Nature Neuroscience.

[42]  Jeffrey B. Henriques,et al.  Decreased responsiveness to reward in depression , 2000 .

[43]  M. Ansseau,et al.  Role of dopamine in non-depressed patients with a history of suicide attempts , 2001, European Psychiatry.

[44]  A. Kelley,et al.  Serotonin-Dopamine Interactions in the Control of Conditioned Reinforcement and Motor Behavior , 2001, Neuropsychopharmacology.

[45]  B. Everitt,et al.  Differential Involvement of NMDA, AMPA/Kainate, and Dopamine Receptors in the Nucleus Accumbens Core in the Acquisition and Performance of Pavlovian Approach Behavior , 2001, The Journal of Neuroscience.

[46]  W. Schultz,et al.  Dopamine responses comply with basic assumptions of formal learning theory , 2001, Nature.

[47]  Paul J. Harrison,et al.  Shorter Oxford textbook of psychiatry , 2001 .

[48]  S. Fuchs,et al.  Reduced dopaminergic activity in depressed suicides , 2001, Psychoneuroendocrinology.

[49]  Maneesh Sahani,et al.  How Linear are Auditory Cortical Responses? , 2002, NIPS.

[50]  T. Robbins,et al.  Nucleus accumbens dopamine depletion impairs both acquisition and performance of appetitive Pavlovian approach behaviour: implications for mesoaccumbens dopamine function , 2002, Behavioural Brain Research.

[51]  C. Naranjo,et al.  Probing brain reward system function in major depressive disorder: altered response to dextroamphetamine. , 2002, Archives of general psychiatry.

[52]  A. Phillips,et al.  A 'crash' course on psychostimulant withdrawal as a model of depression. , 2002, Trends in pharmacological sciences.

[53]  B. Balleine,et al.  The Role of Learning in the Operation of Motivational Systems , 2002 .

[54]  Karl J. Friston,et al.  Temporal Difference Models and Reward-Related Learning in the Human Brain , 2003, Neuron.

[55]  J. Os,et al.  Emotional reactivity to daily life stress in psychosis and affective disorder: an experience sampling study , 2003, Acta psychiatrica Scandinavica.

[56]  A. Grace,et al.  Afferent modulation of dopamine neuron firing differentially regulates tonic and phasic dopamine transmission , 2003, Nature Neuroscience.

[57]  Tatsuo K Sato,et al.  Correlated Coding of Motivation and Outcome of Decision by Dopamine Neurons , 2003, The Journal of Neuroscience.

[58]  F. Gonon,et al.  Presynaptic regulation of dopaminergic neurotransmission , 2003, Journal of neurochemistry.

[59]  Klaus P. Ebmeier,et al.  Neural predictive error signal correlates with depressive illness severity in a game paradigm , 2004, NeuroImage.

[60]  P. Willner,et al.  Reduction of sucrose preference by chronic unpredictable mild stress, and its restoration by a tricyclic antidepressant , 2004, Psychopharmacology.

[61]  C. Nemeroff The Biochemical Basis of Neuropharmacology, 8th ed. , 2004 .

[62]  S. Kapur,et al.  A Model of Antipsychotic Action in Conditioned Avoidance: A Computational Approach , 2004, Neuropsychopharmacology.

[63]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[64]  A. Grace,et al.  Dopaminergic modulation of limbic and cortical drive of nucleus accumbens in goal-directed behavior , 2005, Nature Neuroscience.

[65]  Jerry Nedelman,et al.  Book review: “Bayesian Data Analysis,” Second Edition by A. Gelman, J.B. Carlin, H.S. Stern, and D.B. Rubin Chapman & Hall/CRC, 2004 , 2005, Comput. Stat..

[66]  K. Berridge,et al.  Hedonic Hot Spot in Nucleus Accumbens Shell: Where Do μ-Opioids Cause Increased Hedonic Impact of Sweetness? , 2005, The Journal of Neuroscience.

[67]  Angela J. Yu,et al.  Uncertainty, Neuromodulation, and Attention , 2005, Neuron.

[68]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[69]  D. Pizzagalli,et al.  Toward an objective characterization of an anhedonic phenotype: A signal-detection approach , 2005, Biological Psychiatry.

[70]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[71]  P. Glimcher,et al.  Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal , 2005, Neuron.

[72]  M. Papp,et al.  Parallel changes in dopamine D2 receptor binding in limbic forebrain associated with chronic mild stress-induced anhedonia and its reversal by imipramine , 1994, Psychopharmacology.

[73]  Kyle S. Smith,et al.  The Ventral Pallidum and Hedonic Reward: Neurochemical Maps of Sucrose “Liking” and Food Intake , 2005, The Journal of Neuroscience.

[74]  Jonathan D. Cohen,et al.  Adaptive gain and the role of the locus coeruleus–norepinephrine system in optimal performance , 2005, The Journal of comparative neurology.

[75]  D. Nutt The role of dopamine and norepinephrine in depression and antidepressant treatment. , 2006, The Journal of clinical psychiatry.

[76]  C. Bradshaw,et al.  Comparison of pramipexole and modafinil on arousal, autonomic, and endocrine functions in healthy volunteers , 2006, Journal of psychopharmacology.

[77]  D. Pizzagalli,et al.  Acute Stress Reduces Reward Responsiveness: Implications for Depression , 2006, Biological Psychiatry.

[78]  O. John,et al.  Anticipatory and consummatory components of the experience of pleasure: A scale development study , 2006 .

[79]  Michael J. Frank,et al.  A mechanistic account of striatal dopamine function in human cognition: psychopharmacological studies with cabergoline and haloperidol. , 2006, Behavioral neuroscience.

[80]  W. Hauber,et al.  Inactivation of the ventral tegmental area abolished the general excitatory influence of Pavlovian cues on instrumental performance. , 2006, Learning & memory.

[81]  C. Bradshaw,et al.  Comparison of pramipexole and amisulpride on alertness, autonomic and endocrine functions in healthy volunteers , 2006, Psychopharmacology.

[82]  E. Nestler,et al.  The Mesolimbic Dopamine Reward Circuit in Depression , 2006, Biological Psychiatry.

[83]  P. Dayan,et al.  Tonic dopamine: opportunity costs and the control of response vigor , 2007, Psychopharmacology.

[84]  Eric J. Nestler,et al.  New approaches to antidepressant drug discovery: beyond monoamines , 2006, Nature Reviews Neuroscience.

[85]  Robert A Koeppe,et al.  Dysregulation of endogenous opioid emotion regulation circuitry in major depression in women. , 2006, Archives of general psychiatry.

[86]  Scott J. Russo,et al.  Molecular Adaptations Underlying Susceptibility and Resistance to Social Defeat in Brain Reward Regions , 2007, Cell.

[87]  Q. Huys Reinforcers and control : towards a computational aetiology of depression , 2007 .

[88]  Klaus P. Ebmeier,et al.  Blunted response to feedback information in depressive illness. , 2007, Brain : a journal of neurology.

[89]  C. Nemeroff,et al.  The role of dopamine in the pathophysiology of depression. , 2007, Archives of general psychiatry.

[90]  C. Bradshaw,et al.  Comparison of pramipexole with and without domperidone co-administration on alertness, autonomic, and endocrine functions in healthy volunteers. , 2007, British journal of clinical pharmacology.

[91]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[92]  Brian A. Nosek,et al.  Explicit and implicit cognition: a preliminary test of a dual-process theory of cognitive vulnerability to depression. , 2007, Behaviour research and therapy.

[93]  G. Parker,et al.  Defining melancholia: the primacy of psychomotor disturbance , 2007, Acta psychiatrica Scandinavica. Supplementum.

[94]  Petra E. Pajtas,et al.  Single dose of a dopamine agonist impairs reinforcement learning in humans: Behavioral evidence from a laboratory-based measure of reward responsiveness , 2008, Psychopharmacology.

[95]  H. Manji,et al.  Novel Drugs and Therapeutic Targets for Severe Mood Disorders , 2008, Neuropsychopharmacology.

[96]  M. Fava,et al.  Reduced hedonic capacity in major depressive disorder: evidence from a probabilistic reward task. , 2008, Journal of psychiatric research.

[97]  M. Milders,et al.  Abnormal Temporal Difference Reward-learning Signals in Major Depression Department of Radiology And , 2022 .

[98]  Elena Goetz,et al.  Euthymic Patients with Bipolar Disorder Show Decreased Reward Learning in a Probabilistic Reward Task , 2008, Biological Psychiatry.

[99]  Brian Knutson,et al.  Neural Responses to Monetary Incentives in Major Depression , 2008, Biological Psychiatry.

[100]  Samuel M. McClure,et al.  BOLD Responses Reflecting Dopaminergic Signals in the Human Ventral Tegmental Area , 2008, Science.

[101]  Peter Dayan,et al.  Psychiatry: Insights into depression through normative decision-making models , 2008, NIPS.

[102]  Michael X. Cohen,et al.  Deep Brain Stimulation to Reward Circuitry Alleviates Anhedonia in Refractory Major Depression , 2008, Neuropsychopharmacology.

[103]  Lauren M. Bylsma,et al.  A meta-analysis of emotional reactivity in major depressive disorder. , 2008, Clinical psychology review.

[104]  Karl J. Friston,et al.  Bayesian model selection for group studies , 2009, NeuroImage.

[105]  Jeffrey L. Birk,et al.  Reduced caudate and nucleus accumbens response to rewards in unmedicated individuals with major depressive disorder. , 2009, The American journal of psychiatry.

[106]  Karl J. Friston,et al.  Bayesian model selection for group studies (vol 46, pg 1005, 2009) , 2009 .

[107]  M. Ernst,et al.  fMRI of alterations in reward selection, anticipation, and feedback in major depressive disorder. , 2009, Journal of affective disorders.

[108]  T. Robbins,et al.  Approach and avoidance learning in patients with major depression and healthy controls: relation to anhedonia , 2009, Psychological Medicine.

[109]  Michael J. Frank,et al.  Single dose of a dopamine agonist impairs reinforcement learning in humans: Evidence from event‐related potentials and computational modeling of striatal‐cortical function , 2009, Human brain mapping.

[110]  K. Berridge ‘Liking’ and ‘wanting’ food rewards: Brain substrates and roles in eating disorders , 2009, Physiology & Behavior.

[111]  J. Salamone,et al.  Dopamine, Behavioral Economics, and Effort , 2009, Front. Behav. Neurosci..

[112]  Deanna L. Wallace,et al.  ΔFosB in brain reward circuits mediates resilience to stress and antidepressant responses , 2010, Nature Neuroscience.

[113]  Michael X. Cohen,et al.  Nucleus Accumbens Deep Brain Stimulation Decreases Ratings of Depression and Anxiety in Treatment-Resistant Depression , 2010, Biological Psychiatry.

[114]  J. Roiser,et al.  Reward and Punishment Processing in Depression , 2010, Biological Psychiatry.

[115]  W. Hauber,et al.  The role of nucleus accumbens dopamine in outcome encoding in instrumental and Pavlovian conditioning , 2010, Neurobiology of Learning and Memory.

[116]  R. Gallop,et al.  Unipolar depression does not moderate responses to the Sweet Taste Test , 2010, Depression and anxiety.

[117]  E T Bullmore,et al.  Paradoxical enhancement of choice reaction time performance in patients with major depression , 2010, Journal of psychopharmacology.

[118]  Ronald C Kessler,et al.  Mental disorders as risk factors for substance use, abuse and dependence: results from the 10-year follow-up of the National Comorbidity Survey. , 2010, Addiction.

[119]  G. Jenkins,et al.  Association of mu-opioid receptor variants and response to citalopram treatment in major depressive disorder. , 2010, The American journal of psychiatry.

[120]  X. Zhuang,et al.  Faculty Opinions recommendation of A selective role for dopamine in stimulus-reward learning. , 2010 .

[121]  Brandon L. Warren,et al.  Short- and Long-Term Functional Consequences of Fluoxetine Exposure During Adolescence in Male Rats , 2010, Biological Psychiatry.

[122]  Raymond J. Dolan,et al.  Disentangling the Roles of Approach, Activation and Valence in Instrumental and Pavlovian Responding , 2011, PLoS Comput. Biol..

[123]  D. Pizzagalli,et al.  Corticotropin-Releasing Hormone Receptor Type 1 (CRHR1) Genetic Variation and Stress Interact to Influence Reward Learning , 2011, The Journal of Neuroscience.

[124]  T. Robbins,et al.  Decision Making, Affect, and Learning: Attention and Performance XXIII , 2011 .

[125]  Michael Moutoussis,et al.  Are computational models of any use to psychiatry? , 2011, Neural Networks.

[126]  P. Dayan,et al.  Behavioral/systems/cognitive Action Dominates Valence in Anticipatory Representations in the Human Striatum and Dopaminergic Midbrain , 2010 .

[127]  D. Zald,et al.  Reconsidering anhedonia in depression: Lessons from translational neuroscience , 2011, Neuroscience & Biobehavioral Reviews.

[128]  Lauren M. Bylsma,et al.  Emotional reactivity to daily events in major and minor depression. , 2011, Journal of abnormal psychology.

[129]  Peter Dayan,et al.  Bonsai Trees in Your Head: How the Pavlovian System Sculpts Goal-Directed Choices by Pruning Decision Trees , 2012, PLoS Comput. Biol..

[130]  Janet B W Williams,et al.  Diagnostic and Statistical Manual of Mental Disorders , 2013 .

[131]  P. Dayan,et al.  Dopamine restores reward prediction errors in old age , 2013, Nature Neuroscience.