Translational tests involving non-reward : methodological considerations Journal Item

This review is concerned with methods for assessing the processing of unrewarded responses in experimental animals and the mechanisms underlying performance of these tasks. A number of clinical populations, including Parkinson’s disease, depression, compulsive disorders, and schizophrenia demonstrate either abnormal processing or learning from non-rewarded responses in laboratory-based reinforcement learning tasks. These effects are hypothesized to result from disturbances in modulatory neurotransmitter systems, including dopamine and serotonin. Parallel work in experimental animals has revealed consistent behavioral patterns associated with non-reward and, consistent with the human literature, modulatory roles for specific neurotransmitters. Classical tests involving an important reward omission component include appetitive extinction, ratio schedules of responding, reversal learning, and delay and probability discounting procedures. In addition, innovative behavioral tests have recently been developed leverage probabilistic feedback to specifically assay accommodation of, and learning from, non-rewarded responses. These procedures will be described and reviewed with discussion of the behavioral and neural determinants of performance. A final section focusses specifically on the benefits of trial-by-trial analysis of responding during such tasks, and the implications of such analyses for the translation of findings to clinical studies.

[1]  T. Robbins,et al.  Cognitive inflexibility after prefrontal serotonin depletion is behaviorally and neurochemically specific. , 2006, Cerebral cortex.

[2]  Joseph J. Paton,et al.  Midbrain dopamine neurons control judgment of time , 2016, Science.

[3]  B. Balleine,et al.  A specific role for posterior dorsolateral striatum in human habit learning , 2009, The European journal of neuroscience.

[4]  T. Robbins,et al.  The role of 5-HT2C receptors in touchscreen visual reversal learning in the rat: a cross-site study , 2015, Psychopharmacology.

[5]  Young T. Hong,et al.  Orbitofrontal Dopamine Depletion Upregulates Caudate Dopamine and Alters Behavior via Changes in Reinforcement Sensitivity , 2014, The Journal of Neuroscience.

[6]  J. Salamone Different effects of haloperidol and extinction on instrumental behaviours , 2004, Psychopharmacology.

[7]  J. Jentsch,et al.  Reversal learning as a measure of impulsive and compulsive behavior in addictions , 2011, Psychopharmacology.

[8]  T. Robbins,et al.  Neuropsychological impairment in patients with major depressive disorder: the effects of feedback on task performance , 2003, Psychological Medicine.

[9]  Christopher D. Adams,et al.  Instrumental Responding following Reinforcer Devaluation , 1981 .

[10]  W. Schultz,et al.  Coding of Predicted Reward Omission by Dopamine Neurons in a Conditioned Inhibition Paradigm , 2003, The Journal of Neuroscience.

[11]  Christopher A Krebs,et al.  Preference reversals and effects of D-amphetamine on delay discounting in rats , 2012, Behavioural pharmacology.

[12]  P. Killeen,et al.  A theory of behaviour on progressive ratio schedules, with applications in behavioural pharmacology , 2012, Psychopharmacology.

[13]  T. Robbins,et al.  Markers of Serotonergic Function in the Orbitofrontal Cortex and Dorsal Raphé Nucleus Predict Individual Variation in Spatial-Discrimination Serial Reversal Learning , 2015, Neuropsychopharmacology.

[14]  Talia N. Lerner,et al.  Nucleus accumbens D2R cells signal prior outcomes and control risky decision-making , 2016, Nature.

[15]  Ryan D Ward,et al.  Pharmacologic Rescue of Motivational Deficit in an Animal Model of the Negative Symptoms of Schizophrenia , 2011, Biological Psychiatry.

[16]  Ryan D Ward,et al.  Modeling motivational deficits in mouse models of schizophrenia: Behavior analysis as a guide for neuroscience , 2011, Behavioural Processes.

[17]  Stan B. Floresco,et al.  Contributions of the nucleus accumbens and its subregions to different aspects of risk-based decision making , 2011, Cognitive, affective & behavioral neuroscience.

[18]  S. Floresco,et al.  Fundamental Contribution by the Basolateral Amygdala to Different Forms of Decision Making , 2009, The Journal of Neuroscience.

[19]  Rudolf N. Cardinal,et al.  Neural systems implicated in delayed and probabilistic reinforcement , 2006, Neural Networks.

[20]  T. Robbins,et al.  Enhanced behavioural control by conditioned reinforcers following microinjections of d-amphetamine into the nucleus accumbens , 2004, Psychopharmacology.

[21]  T. Robbins,et al.  Abnormal response to negative feedback in unipolar depression: evidence for a diagnosis specific impairment , 1997, Journal of neurology, neurosurgery, and psychiatry.

[22]  B. Franke,et al.  Dissociable Effects of Dopamine and Serotonin on Reversal Learning , 2013, Neuron.

[23]  M. Bear,et al.  Extinction of an instrumental response: a cognitive behavioral assay in Fmr1 knockout mice , 2014, Genes, brain, and behavior.

[24]  P. Dayan,et al.  States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning , 2010, Neuron.

[25]  L. Saksida,et al.  Paradoxical reversal learning enhancement by stress or prefrontal cortical damage: rescue with BDNF , 2011, Nature Neuroscience.

[26]  L. Saksida,et al.  The touchscreen cognitive testing method for rodents: how to get the best out of your rat. , 2008, Learning & memory.

[27]  Role of ionotropic glutamate receptors in delay and probability discounting in the rat , 2015, Psychopharmacology.

[28]  Sham M. Kakade,et al.  Opponent interactions between serotonin and dopamine , 2002, Neural Networks.

[29]  Richard D Emes,et al.  Synaptic scaffold evolution generated components of vertebrate cognitive complexity , 2012, Nature Neuroscience.

[30]  R. Elliott,et al.  Temporal discounting in major depressive disorder , 2013, Psychological Medicine.

[31]  L Miller,et al.  Neuropsychodynamics of alcoholism and addiction: personality, psychopathology, and cognitive style. , 1990, Journal of substance abuse treatment.

[32]  L. Saksida,et al.  Pharmacological or genetic inactivation of the serotonin transporter improves reversal learning in mice. , 2010, Cerebral cortex.

[33]  B. Sahakian,et al.  The neuropsychology of mood disorders , 2005, Current psychiatry reports.

[34]  B. Moghaddam,et al.  Adaptive Encoding of Outcome Prediction by Prefrontal Cortex Ensembles Supports Behavioral Flexibility , 2017, The Journal of Neuroscience.

[35]  Samuel M. McClure,et al.  Separate Neural Systems Value Immediate and Delayed Monetary Rewards , 2004, Science.

[36]  T. Robbins,et al.  Serotonin Modulates Sensitivity to Reward and Negative Feedback in a Probabilistic Reversal Learning Task in Rats , 2010, Neuropsychopharmacology.

[37]  J. Salamone,et al.  Effects of Dopamine Antagonists and Accumbens Dopamine Depletions on Time-Constrained Progressive-Ratio Performance , 1998, Pharmacology Biochemistry and Behavior.

[38]  R. Lubow,et al.  The abolition of the partial reinforcement extinction effect (PREE) by amphetamine , 2004, Psychopharmacology.

[39]  Stephan F. Miedl,et al.  Altered neural reward representations in pathological gamblers revealed by delay and probability discounting. , 2012, Archives of general psychiatry.

[40]  L. Saksida,et al.  Impaired discrimination learning in mice lacking the NMDA receptor NR2A subunit. , 2008, Learning & memory.

[41]  R. Joosten,et al.  Phasic dopamine release induced by positive feedback predicts individual differences in reversal learning , 2015, Neurobiology of Learning and Memory.

[42]  G. Strauss,et al.  Avolition in schizophrenia is associated with reduced willingness to expend effort for reward on a Progressive Ratio task , 2016, Schizophrenia Research.

[43]  Midbrain Dopamine Neurons and Adult Neurogenesis , 2012 .

[44]  Lisa M. Wiedholz,et al.  Do GluA1 knockout mice exhibit behavioral abnormalities relevant to the negative or cognitive symptoms of schizophrenia and schizoaffective disorder? , 2012, Neuropharmacology.

[45]  P. Killeen Mathematical principles of reinforcement , 1994 .

[46]  D. Pizzagalli,et al.  Reduced Reward Learning Predicts Outcome in Major Depressive Disorder , 2013, Biological Psychiatry.

[47]  B. Balleine,et al.  Goal-directed instrumental action: contingency and incentive learning and their cortical substrates , 1998, Neuropharmacology.

[48]  K. Lesch,et al.  Establishing a probabilistic reversal learning test in mice: Evidence for the processes mediating reward-stay and punishment-shift behaviour and for their modulation by serotonin , 2012, Neuropharmacology.

[49]  R. Wise,et al.  Neuroleptic-induced "anhedonia" in rats: pimozide blocks reward quality of food. , 1978, Science.

[50]  K. Mueser,et al.  Assessment of enduring deficit and negative symptom subtypes in schizophrenia. , 1991, Schizophrenia bulletin.

[51]  G. Alexopoulos,et al.  Reward learning impairment and avoidance and rumination responses at the end of Engage therapy of late‐life depression , 2018, International journal of geriatric psychiatry.

[52]  S. Mitchell Assessing Delay Discounting in Mice , 2014, Current protocols in neuroscience.

[53]  Michael J. Frank,et al.  By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism , 2004, Science.

[54]  Deanna M Barch,et al.  Probabilistic Reinforcement Learning in Patients With Schizophrenia: Relationships to Anhedonia and Avolition. , 2016, Biological psychiatry. Cognitive neuroscience and neuroimaging.

[55]  T. Robbins,et al.  Cognitive Inflexibility After Prefrontal Serotonin Depletion , 2004, Science.

[56]  T. Robbins,et al.  Approach and avoidance learning in patients with major depression and healthy controls: relation to anhedonia , 2009, Psychological Medicine.

[57]  Effects of amphetamine and methylphenidate on delay discounting in rats: interactions with order of delay presentation , 2013, Psychopharmacology.

[58]  J. Salamone,et al.  Effort-Related Motivational Effects of the VMAT-2 Inhibitor Tetrabenazine: Implications for Animal Models of the Motivational Symptoms of Depression , 2013, The Journal of Neuroscience.

[59]  B. Balleine,et al.  The Effect of Lesions of the Insular Cortex on Instrumental Conditioning: Evidence for a Role in Incentive Memory , 2000, The Journal of Neuroscience.

[60]  J. Salamone,et al.  Dopaminergic Modulation of Effort-Related Choice Behavior as Assessed by a Progressive Ratio Chow Feeding Choice Task: Pharmacological Studies and the Role of Individual Differences , 2012, PloS one.

[61]  L. Saksida,et al.  Selective effects of 5-HT2C receptor modulation on performance of a novel valence-probe visual discrimination task and probabilistic reversal learning in mice , 2018, Psychopharmacology.

[62]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[63]  R. Stark,et al.  Neural correlates of appetitive extinction in humans , 2016, Social cognitive and affective neuroscience.

[64]  G. Light,et al.  Relationship between effortful motivation and neurocognition in schizophrenia , 2017, Schizophrenia Research.

[65]  J. Yates,et al.  Effects of GluN2B-selective Antagonists on Delay and Probability Discounting in Male Rats: Modulation by Delay/Probability Presentation Order , 2018, Experimental and clinical psychopharmacology.

[66]  T. Robbins,et al.  Dissociable Effects of Selective 5-HT2A and 5-HT2C Receptor Antagonists on Serial Spatial Reversal Learning in Rats , 2008, Neuropsychopharmacology.

[67]  L. Saksida,et al.  Motivational assessment of mice using the touchscreen operant testing system: effects of dopaminergic drugs , 2015, Psychopharmacology.

[68]  M. Bouton Context and behavioral processes in extinction. , 2004, Learning & memory.

[69]  T. Robbins,et al.  Prefrontal Serotonin Depletion Affects Reversal Learning But Not Attentional Set Shifting , 2005, The Journal of Neuroscience.

[70]  S. Mitchell,et al.  Impact of strain and d-amphetamine on impulsivity (delay discounting) in inbred mice , 2006, Psychopharmacology.

[71]  T. Robbins,et al.  Serotonergic Modulation of Prefrontal Cortex during Negative Feedback in Probabilistic Reversal Learning , 2005, Neuropsychopharmacology.

[72]  G. Higa,et al.  Serotonergic modulation of prefrontal cortex plasticity: Role of 5HT1A in a depression animal model , 2019, IBRO Reports.

[73]  Hongli Zhou,et al.  Impaired risk evaluation in people with Internet gaming disorder: fMRI evidence from a probability discounting task , 2015, Progress in Neuro-Psychopharmacology and Biological Psychiatry.

[74]  Joshua D. Larkin,et al.  Prefrontal Dopamine D1 and D2 Receptors Regulate Dissociable Aspects of Decision Making via Distinct Ventral Striatal and Amygdalar Circuits , 2017, The Journal of Neuroscience.

[75]  Theresia J. M. Roelofs,et al.  A neuronal mechanism underlying decision-making deficits during hyperdopaminergic states , 2017, Nature Communications.

[76]  M. Sidman,et al.  Satiation effects under fixed-ratio schedules of reinforcement. , 1954, Journal of comparative and physiological psychology.

[77]  Madalena S. Fonseca,et al.  An effect of serotonergic stimulation on learning rates for rewards apparent after long intertrial intervals , 2018, Nature Communications.

[78]  Johan Alsiö,et al.  The rat's not for turning: Dissociating the psychological components of cognitive inflexibility☆ , 2015, Neuroscience & Biobehavioral Reviews.

[79]  T. Robbins,et al.  L-DOPA Disrupts Activity in the Nucleus Accumbens during Reversal Learning in Parkinson's Disease , 2007, Neuropsychopharmacology.

[80]  D. Clair,et al.  Bridging the translational divide: identical cognitive touchscreen testing in mice and humans carrying mutations in a disease-relevant homologous gene , 2015, Scientific Reports.

[81]  L. Saksida,et al.  GluN2B in corticostriatal circuits governs choice learning and choice shifting , 2013, Nature Neuroscience.

[82]  Brian M. Sweis,et al.  Mice learn to avoid regret , 2018, PLoS biology.

[83]  H. Niki,et al.  Food‐Reinforced Operant Behavior in Dopamine Transporter Knockout Mice: Enhanced Resistance to Extinction , 2004, Annals of the New York Academy of Sciences.

[84]  Y. Chudasama,et al.  Dissociable contributions of the ventral hippocampus and orbitofrontal cortex to decision‐making with a delayed or uncertain outcome , 2013, The European journal of neuroscience.

[85]  Justin K. O’Hare,et al.  Striatal fast-spiking interneurons selectively modulate circuit output and are required for habitual behavior , 2017, eLife.

[86]  L. Saksida,et al.  Enhanced cognition and dysregulated hippocampal synaptic physiology in mice with a heterozygous deletion of PSD‐95 , 2018, The European journal of neuroscience.

[87]  M. Bouton,et al.  Separation of time-based and trial-based accounts of the partial reinforcement extinction effect , 2014, Behavioural Processes.

[88]  Warren K. Bickel,et al.  Discounting of Delayed Rewards as an Endophenotype , 2015, Biological Psychiatry.

[89]  B. Jelfs,et al.  Impairment of decision making and disruption of synchrony between basolateral amygdala and anterior cingulate cortex in the maternally separated rat , 2016, Neurobiology of Learning and Memory.

[90]  Derek G. V. Mitchell,et al.  Dissociable roles of medial orbitofrontal cortex in human operant extinction learning , 2008, NeuroImage.

[91]  J. Salamone,et al.  Selection of sucrose concentration depends on the effort required to obtain it: studies using tetrabenazine, D1, D2, and D3 receptor antagonists , 2015, Psychopharmacology.

[92]  W HODOS,et al.  Progressive Ratio as a Measure of Reward Strength , 1961, Science.

[93]  E. T. Bullmore,et al.  Reinforcement and Reversal Learning in First-Episode Psychosis , 2008, Schizophrenia bulletin.

[94]  Nicole A. Crowley,et al.  Chronic alcohol produces neuroadaptations to prime dorsal striatal learning , 2013, Proceedings of the National Academy of Sciences.

[95]  Jeffrey S. Stein,et al.  Delay discounting and gambling , 2011, Behavioural Processes.

[96]  J. Brigman,et al.  Touch-screen visual reversal learning is mediated by value encoding and signal propagation in the orbitofrontal cortex , 2017, Neurobiology of Learning and Memory.

[97]  J. Evenden,et al.  The pharmacology of impulsive behaviour in rats: the effects of drugs on response choice with varying delays of reinforcement , 1996, Psychopharmacology.

[98]  Benjamin M. Robinson,et al.  Selective Reinforcement Learning Deficits in Schizophrenia Support Predictions from Computational Models of Striatal-Cortical Dysfunction , 2007, Biological Psychiatry.

[99]  L. Saksida,et al.  Optimisation of cognitive performance in rodent operant (touchscreen) testing: Evaluation and effects of reinforcer strength , 2017, Learning & behavior.

[100]  William W. Taylor,et al.  Dorsolateral Striatum Engagement Interferes with Early Discrimination Learning , 2018, Cell reports.

[101]  U. Bromberg,et al.  The Role of Prospection in Steep Temporal Reward Discounting in Gambling Addiction , 2015, Front. Psychiatry.

[102]  P. Janak,et al.  Habitual Alcohol Seeking: Time Course and the Contribution of Subregions of the Dorsal Striatum , 2012, Biological Psychiatry.

[103]  A. Dickinson,et al.  Differential Engagement of the Ventromedial Prefrontal Cortex by Goal-Directed and Habitual Behavior toward Food Pictures in Humans , 2009, The Journal of Neuroscience.

[104]  S. Floresco,et al.  Dopaminergic Modulation of Risk-Based Decision Making , 2009, Neuropsychopharmacology.

[105]  R. Rygula,et al.  Ketamine decreases sensitivity of male rats to misleading negative feedback in a probabilistic reversal-learning task , 2016, Psychopharmacology.

[106]  Russell G. Port,et al.  Nucleus accumbens and effort-related functions: behavioral and neural markers of the interactions between adenosine A2A and dopamine D2 receptors , 2010, Neuroscience.

[107]  Relative reinforcing value of exercise in inpatients with anorexia nervosa: model development and pilot data. , 2007, The International journal of eating disorders.