Impairments in reinforcement learning do not explain enhanced habit formation in cocaine use disorder

Drug addiction has been suggested to develop through drug-induced changes in learning and memory processes. Whilst the initiation of drug use is typically goal-directed and hedonically motivated, over time, drug-taking may develop into a stimulus-driven habit, characterised by persistent use of the drug irrespective of the consequences. Converging lines of evidence suggest that stimulant drugs facilitate the transition of goal-directed into habitual drug-taking, but their contribution to goal-directed learning is less clear. Computational modelling may provide an elegant means for elucidating changes during instrumental learning that may explain enhanced habit formation. We used formal reinforcement learning algorithms to deconstruct the process of appetitive instrumental learning and to explore potential associations between goal-directed and habitual actions in patients with cocaine use disorder (CUD). We re-analysed appetitive instrumental learning data in 55 healthy control volunteers and 70 CUD patients by applying a reinforcement learning model within a hierarchical Bayesian framework. We used a regression model to determine the influence of learning parameters and variations in brain structure on subsequent habit formation. Poor instrumental learning performance in CUD patients was largely determined by difficulties with learning from feedback, as reflected by a significantly reduced learning rate. Subsequent formation of habitual response patterns was partly explained by group status and individual variation in reinforcement sensitivity. White matter integrity within goal-directed networks was only associated with performance parameters in controls but not in CUD patients. Our data indicate that impairments in reinforcement learning are insufficient to account for enhanced habitual responding in CUD.

[1]  Christopher D. Adams,et al.  Instrumental Responding following Reinforcer Devaluation , 1981 .

[2]  A. Dickinson Actions and habits: the development of behavioural autonomy , 1985 .

[3]  O. Aasland,et al.  Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO Collaborative Project on Early Detection of Persons with Harmful Alcohol Consumption--II. , 1993, Addiction.

[4]  N. Volkow,et al.  Decreased dopamine D2 receptor availability is associated with reduced frontal metabolism in cocaine abusers , 1993, Synapse.

[5]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[6]  D. Dougherty,et al.  Laboratory measurement of adaptive behavior change in humans with a history of substance dependence. , 1998, Drug and alcohol dependence.

[7]  Andrew Gelman,et al.  General methods for monitoring convergence of iterative simulations , 1998 .

[8]  D. Sheehan,et al.  The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. , 1998, The Journal of clinical psychiatry.

[9]  松岡 恵子,et al.  日本語版National Adult Reading Test(JART)の作成 , 2002 .

[10]  M. First,et al.  Structured Clinical Interview for DSM-IV-TR Axis I Disorders, Research version (SCID-I RV) , 2002 .

[11]  A. Dickinson,et al.  Oral cocaine seeking by rats: action or habit? , 2003, Behavioral neuroscience.

[12]  B. Everitt,et al.  Drug Seeking Becomes Compulsive After Prolonged Cocaine Self-Administration , 2004, Science.

[13]  Seth J. Ramus,et al.  Cocaine‐experienced rats exhibit learning deficits in a task sensitive to orbitofrontal cortex lesions , 2004, The European journal of neuroscience.

[14]  T. Robbins,et al.  Neural systems of reinforcement for drug addiction: from actions to habits to compulsion , 2005, Nature Neuroscience.

[15]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[16]  S. Hyman Addiction: a disease of learning and memory. , 2005, The American journal of psychiatry.

[17]  M. Pérez-García,et al.  Profile of executive deficits in cocaine and heroin polysubstance users: common and differential effects on separate executive components , 2007, Psychopharmacology.

[18]  J. O'Doherty,et al.  Is Avoiding an Aversive Outcome Rewarding? Neural Substrates of Avoidance Learning in the Human Brain , 2006, PLoS biology.

[19]  S. Killcross,et al.  Amphetamine Exposure Enhances Habit Formation , 2006, The Journal of Neuroscience.

[20]  A. Dickinson,et al.  Stimulus-outcome interactions during instrumental discrimination learning by rats and humans. , 2007, Journal of experimental psychology. Animal behavior processes.

[21]  Mark Slifstein,et al.  Amphetamine-induced dopamine release: markedly blunted in cocaine dependence and predictive of the choice to self-administer cocaine. , 2007, The American journal of psychiatry.

[22]  Vivian V. Valentin,et al.  Determining the Neural Substrates of Goal-Directed Learning in the Human Brain , 2007, The Journal of Neuroscience.

[23]  T. Robbins,et al.  Chronic cocaine but not chronic amphetamine use is associated with perseverative responding in humans , 2008, Psychopharmacology.

[24]  M. Roesch,et al.  Cocaine Exposure Shifts the Balance of Associative Encoding from Ventral to Dorsolateral Striatum , 2007, Frontiers in integrative neuroscience.

[25]  Donna J. Calu,et al.  Withdrawal from cocaine self-administration produces long-lasting deficits in orbitofrontal-dependent reversal learning in rats. , 2007, Learning & memory.

[26]  B. Everitt,et al.  Cocaine Seeking Habits Depend upon Dopamine-Dependent Serial Connectivity Linking the Ventral with the Dorsal Striatum , 2008, Neuron.

[27]  B. Balleine,et al.  Calculating Consequences: Brain Systems That Encode the Causal Effects of Actions , 2008, The Journal of Neuroscience.

[28]  B. Balleine,et al.  A specific role for posterior dorsolateral striatum in human habit learning , 2009, The European journal of neuroscience.

[29]  A. Dickinson,et al.  Differential Engagement of the Ventromedial Prefrontal Cortex by Goal-Directed and Habitual Behavior toward Food Pictures in Humans , 2009, The Journal of Neuroscience.

[30]  B. Balleine,et al.  Human and Rodent Homologies in Action Control: Corticostriatal Determinants of Goal-Directed and Habitual Action , 2010, Neuropsychopharmacology.

[31]  John Suckling,et al.  Influence of compulsivity of drug abuse on dopaminergic modulation of attentional bias in stimulant dependence. , 2010, Archives of general psychiatry.

[32]  Nathaniel D. Daw,et al.  Trial-by-trial data analysis using computational models , 2011 .

[33]  Andrea Brovelli,et al.  Differential roles of caudate nucleus and putamen during instrumental learning , 2011, NeuroImage.

[34]  T. Robbins,et al.  Response Perseveration in Stimulant Dependence Is Associated with Striatal Dysfunction and Can Be Ameliorated by a D2/3 Receptor Agonist , 2011, Biological Psychiatry.

[35]  T. Robbins,et al.  Decision Making, Affect, and Learning: Attention and Performance XXIII , 2011 .

[36]  T. Robbins,et al.  Disruption in the Balance Between Goal-Directed Behavior and Habit Learning in Obsessive-Compulsive Disorder , 2011, The American journal of psychiatry.

[37]  Guy B. Williams,et al.  Abnormal Brain Structure Implicated in Stimulant Drug Addiction , 2012, Science.

[38]  K. R. Ridderinkhof,et al.  Corticostriatal Connectivity Underlies Individual Differences in the Balance between Habitual and Goal-Directed Action Control , 2012, The Journal of Neuroscience.

[39]  T. Robbins,et al.  Neurocognitive endophenotypes of impulsivity and compulsivity: towards dimensional psychiatry , 2012, Trends in Cognitive Sciences.

[40]  David B. Dunson,et al.  Bayesian data analysis, third edition , 2013 .

[41]  Michael J. Brammer,et al.  Neural and Psychological Maturation of Decision-making in Adolescence and Young Adulthood , 2013, Journal of Cognitive Neuroscience.

[42]  JaneR . Taylor,et al.  Cytoskeletal Determinants of Stimulus-Response Habits , 2013, The Journal of Neuroscience.

[43]  T. Robbins,et al.  Behavioral and neuroimaging evidence for overreliance on habit learning in alcohol-dependent patients , 2013, Translational Psychiatry.

[44]  Jody Tanabe,et al.  Reduced neural tracking of prediction error in substance-dependent individuals. , 2013, The American journal of psychiatry.

[45]  John J. Foxe,et al.  The influence of monetary punishment on cognitive control in abstinent cocaine-users. , 2013, Drug and alcohol dependence.

[46]  B. Balleine,et al.  Effects of Repeated Cocaine Exposure on Habit Learning and Reversal by N-Acetylcysteine , 2014, Neuropsychopharmacology.

[47]  T. Vos,et al.  The global epidemiology and burden of psychostimulant dependence: findings from the Global Burden of Disease Study 2010. , 2014, Drug and alcohol dependence.

[48]  G. Arbanas Diagnostic and Statistical Manual of Mental Disorders (DSM-5) , 2015 .

[49]  John J. Foxe,et al.  Regulating task-monitoring systems in response to variable reward contingencies and outcomes in cocaine addicts , 2016, Psychopharmacology.

[50]  G. Schoenbaum,et al.  Effects of Prior Cocaine Versus Morphine or Heroin Self-Administration on Extinction Learning Driven by Overexpectation Versus Omission of Reward , 2015, Biological Psychiatry.

[51]  Y. Niv,et al.  Temporal Specificity of Reward Prediction Errors Signaled by Putative Dopamine Neurons in Rat VTA Depends on Ventral Striatum , 2016, Neuron.

[52]  A. Heinz,et al.  Dimensional psychiatry: mental disorders as dysfunctions of basic learning mechanisms , 2016, Journal of Neural Transmission.

[53]  Guy B. Williams,et al.  Carrots and sticks fail to change behavior in cocaine addiction , 2016, Science.

[54]  Charlotte A. Boettiger,et al.  Addiction History Associates with the Propensity to Form Habits , 2016, Journal of Cognitive Neuroscience.

[55]  W. Stoops,et al.  Differential sensitivity to learning from positive and negative outcomes in cocaine users. , 2016, Drug and alcohol dependence.

[56]  T. Robbins,et al.  Drug Addiction: Updating Actions to Habits to Compulsions Ten Years On. , 2016, Annual review of psychology.

[57]  S. Gershman Empirical priors for reinforcement learning models , 2016 .

[58]  R. Costa,et al.  Habits , 2014 .

[59]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[60]  David S. Leslie,et al.  A tutorial on bridge sampling , 2017, Journal of mathematical psychology.

[61]  P. Janak,et al.  Defining the place of habit in substance use disorders , 2017, Progress in Neuro-Psychopharmacology and Biological Psychiatry.

[62]  Hannes Ruge,et al.  Habit strength is predicted by activity dynamics in goal-directed brain systems during training , 2018, NeuroImage.

[63]  G. Schoenbaum,et al.  Expectancy-Related Changes in Dopaminergic Error Signals Are Impaired by Cocaine Self-Administration , 2019, Neuron.

[64]  T. Robbins,et al.  Goal-Directed and Habitual Control in Smokers , 2019, Nicotine & tobacco research : official journal of the Society for Research on Nicotine and Tobacco.