Catecholaminergic Regulation of Learning Rate in a Dynamic Environment

Adaptive behavior in a changing world requires flexibly adapting one’s rate of learning to the rate of environmental change. Recent studies have examined the computational mechanisms by which various environmental factors determine the impact of new outcomes on existing beliefs (i.e., the ‘learning rate’). However, the brain mechanisms, and in particular the neuromodulators, involved in this process are still largely unknown. The brain-wide neurophysiological effects of the catecholamines norepinephrine and dopamine on stimulus-evoked cortical responses suggest that the catecholamine systems are well positioned to regulate learning about environmental change, but more direct evidence for a role of this system is scant. Here, we report evidence from a study employing pharmacology, scalp electrophysiology and computational modeling (N = 32) that suggests an important role for catecholamines in learning rate regulation. We found that the P3 component of the EEG—an electrophysiological index of outcome-evoked phasic catecholamine release in the cortex—predicted learning rate, and formally mediated the effect of prediction-error magnitude on learning rate. P3 amplitude also mediated the effects of two computational variables—capturing the unexpectedness of an outcome and the uncertainty of a preexisting belief—on learning rate. Furthermore, a pharmacological manipulation of catecholamine activity affected learning rate following unanticipated task changes, in a way that depended on participants’ baseline learning rate. Our findings provide converging evidence for a causal role of the human catecholamine systems in learning-rate regulation as a function of environmental change.

[1]  I. Lucki,et al.  Effects of Acute and Chronic Reboxetine Treatment on Stress-induced Monoamine Efflux in the Rat Frontal Cortex , 2002, Neuropsychopharmacology.

[2]  W. Einhäuser,et al.  Pupil Dilation Signals Surprise: Evidence for Noradrenaline’s Role in Decision Making , 2011, Front. Neurosci..

[3]  Therese van Amelsvoort,et al.  The Detection of Novelty Relies on Dopaminergic Signaling: Evidence from Apomorphine's Impact on the Novelty N2 , 2013, PloS one.

[4]  K. Branson,et al.  Behavioral Variability through Stochastic Choice and Its Gating by Anterior Cingulate Cortex , 2014, Cell.

[5]  S. Nieuwenhuis,et al.  The anatomical and functional relationship between the P3 and autonomic components of the orienting response. , 2011, Psychophysiology.

[6]  Sverker Sikström,et al.  Stimulus-dependent dopamine release in attention-deficit/hyperactivity disorder. , 2007, Psychological review.

[7]  Timothy E. J. Behrens,et al.  Dissociable effects of surprise and model update in parietal and anterior cingulate cortex , 2013, Proceedings of the National Academy of Sciences.

[8]  Silvio Garattini,et al.  Role of presynaptic α2-adrenoceptors in antidepressant action: recent findings from microdialysis studies , 2004, Progress in Neuro-Psychopharmacology and Biological Psychiatry.

[9]  J. Pineda,et al.  Effects of locus coeruleus lesions on auditory, long-latency, event- related potentials in monkey , 1989, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[10]  Soo Hong Chew,et al.  Dissociable contribution of prefrontal and striatal dopaminergic genes to learning in economic games , 2014, Proceedings of the National Academy of Sciences.

[11]  G. Aston-Jones,et al.  Atomoxetine modulates spontaneous and sensory-evoked discharge of locus coeruleus noradrenergic neurons , 2013, Neuropharmacology.

[12]  Tom Verguts,et al.  The P3 Event-Related Potential is a Biomarker for the Efficacy of Vagus Nerve Stimulation in Patients with Epilepsy , 2014, Neurotherapeutics.

[13]  E R John,et al.  Information Delivery and the Sensory Evoked Potential , 1967, Science.

[14]  Peter N. C. Mohr,et al.  Genetic variation in dopaminergic neuromodulation influences the ability to rapidly and flexibly adapt decisions , 2009, Proceedings of the National Academy of Sciences.

[15]  E. Donchin Presidential address, 1980. Surprise!...Surprise? , 1981, Psychophysiology.

[16]  T. Price,et al.  Correcting for Regression to the Mean in Behavior and Ecology , 2005, The American Naturalist.

[17]  S. Sara,et al.  Activation of the noradrenergic system facilitates an attentional shift in the rat , 1990, Behavioural Brain Research.

[18]  R. Wise Dopamine, learning and motivation , 2004, Nature Reviews Neuroscience.

[19]  S. Debener,et al.  Trial-by-Trial Fluctuations in the Event-Related Electroencephalogram Reflect Dynamic Changes in the Degree of Surprise , 2008, The Journal of Neuroscience.

[20]  Maurizio Corbetta,et al.  Comment on “Modafinil Shifts Human Locus Coeruleus to Low-Tonic, High-Phasic Activity During Functional MRI” and “Homeostatic Sleep Pressure and Responses to Sustained Attention in the Suprachiasmatic Area” , 2010, Science.

[21]  E. Pitman A NOTE ON NORMAL CORRELATION , 1939 .

[22]  Jyrki Ahveninen,et al.  Dopamine modulates involuntary attention shifting and reorienting: an electromagnetic study , 2002, Clinical Neurophysiology.

[23]  E Donchin,et al.  A new method for off-line removal of ocular artifact. , 1983, Electroencephalography and clinical neurophysiology.

[24]  Joseph T. McGuire,et al.  Functionally Dissociable Influences on Learning Rate in a Dynamic Environment , 2014, Neuron.

[25]  D. Morilak,et al.  Noradrenergic modulation of cognitive function in rat medial prefrontal cortex as measured by attentional set shifting capability , 2006, Neuroscience.

[26]  D. A. Kenny,et al.  Lower Level Mediation in Multilevel Models , 2022 .

[27]  E. Donchin,et al.  Is the P300 component a manifestation of context updating? , 1988, Behavioral and Brain Sciences.

[28]  Tim Fingscheidt,et al.  A Model-Based Approach to Trial-By-Trial P300 Amplitude Fluctuations , 2013, Front. Hum. Neurosci..

[29]  K. Perry,et al.  Atomoxetine Increases Extracellular Levels of Norepinephrine and Dopamine in Prefrontal Cortex of Rat: A Potential Mechanism for Efficacy in Attention Deficit/Hyperactivity Disorder , 2002, Neuropsychopharmacology.

[30]  M. Hansenne,et al.  Le potentiel évoqué cognitif P300 (I) : aspects théorique et psychobiologique , 2000, Neurophysiologie Clinique/Clinical Neurophysiology.

[31]  Sander Nieuwenhuis,et al.  Noradrenergic and cholinergic modulation of late ERP responses to deviant stimuli. , 2015, Psychophysiology.

[32]  A. Gibbs,et al.  Catechol-O-methyltransferase val158met genotype determines effect of reboxetine on emotional memory in healthy male volunteers. , 2014, Journal of psychiatry & neuroscience : JPN.

[33]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 2005, IEEE Transactions on Neural Networks.

[34]  D. Weinberger,et al.  Genes, dopamine and cortical signal-to-noise ratio in schizophrenia , 2004, Trends in Neurosciences.

[35]  Jonathan D. Cohen,et al.  An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. , 2005, Annual review of neuroscience.

[36]  Sander Nieuwenhuis,et al.  Learning, the P3, and the Locus Coeruleus-Norepinephrine System , 2011 .

[37]  S. Sara,et al.  Orienting and Reorienting: The Locus Coeruleus Mediates Cognition through Arousal , 2012, Neuron.

[38]  J. McGaughy,et al.  Atomoxetine reverses attentional deficits produced by noradrenergic deafferentation of medial prefrontal cortex , 2008, Psychopharmacology.

[39]  S. Bouret,et al.  Noradrenaline and Dopamine Neurons in the Reward/Effort Trade-Off: A Direct Electrophysiological Comparison in Behaving Monkeys , 2015, The Journal of Neuroscience.

[40]  P. Goldman-Rakic,et al.  Selective prefrontal cortical projections to the region of the locus coeruleus and raphe nuclei in the rhesus monkey , 1984, Brain Research.

[41]  W. Gerstner,et al.  Stress, genotype and norepinephrine in the prediction of mouse behavior using reinforcement learning , 2009, Nature Neuroscience.

[42]  Nathalie Boot,et al.  Beta receptor-mediated modulation of the late positive potential in humans , 2011, Psychopharmacology.

[43]  Timothy E. J. Behrens,et al.  Anxious individuals have difficulty learning the causal statistics of aversive environments , 2015, Nature Neuroscience.

[44]  G. Gessa,et al.  Alpha2‐adrenoceptor mediated co‐release of dopamine and noradrenaline from noradrenergic neurons in the cerebral cortex , 2004, Journal of neurochemistry.

[45]  Carla M. Bosco,et al.  Event-related brain potentials as indices of information extraction and response priming. , 1990, Electroencephalography and clinical neurophysiology.

[46]  Massimo Silvetti,et al.  Value and prediction error estimation account for volatility effects in ACC: A model-based fMRI study , 2013, Cortex.

[47]  M. McCabe Presidential address. , 1970, Irish nurses' journal.

[48]  H. Möller,et al.  The effect of the skull on event-related P300 , 2001, Clinical Neurophysiology.

[49]  A. Gibbs,et al.  Catechol-O-methyltransferase val 158 met genotype determines effect of reboxetine on emotional memory in healthy male volunteers , 2014 .

[50]  J T Coull Modulation of attention by noradrenergic alpha2-agents varies according to arousal level. , 2001, Drug news & perspectives.

[51]  Martin A. Lindquist,et al.  Brain mediators of cardiovascular responses to social threat Part I: Reciprocal dorsal and ventral sub-regions of the medial prefrontal cortex and heart-rate reactivity , 2009, NeuroImage.

[52]  Raymond J. Dolan,et al.  Learning and Generalization under Ambiguity: An fMRI Study , 2012, PLoS Comput. Biol..

[53]  S. Sara The locus coeruleus and noradrenergic modulation of cognition , 2009, Nature Reviews Neuroscience.

[54]  Coull Jt Modulation of attention by noradrenergic alpha2-agents varies according to arousal level. , 2001 .

[55]  Toshio Matsuda,et al.  Effects of acute and chronic administration of atomoxetine and methylphenidate on extracellular levels of noradrenaline, dopamine and serotonin in the prefrontal cortex and striatum of mice , 2010, Journal of neurochemistry.

[56]  T. Crow,et al.  Impaired Learning and Decreased Cortical Norepinephrine after Bilateral Locus Coeruleus Lesions , 1973, Science.

[57]  Eric-Jan Wagenmakers,et al.  The Role of the Noradrenergic System in the Exploration–Exploitation Trade-Off: A Psychopharmacological Study , 2010, Front. Hum. Neurosci..

[58]  Colin Camerer,et al.  Explicit neural signals reflecting reward uncertainty , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[59]  M. Corbetta,et al.  The Reorienting System of the Human Brain: From Environment to Theory of Mind , 2008, Neuron.

[60]  Jonathan D. Cohen,et al.  Computational roles for dopamine in behavioural control , 2004, Nature.

[61]  T. Robbins,et al.  Chemistry of the adaptive mind , 2004, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[62]  A. Bond,et al.  The use of analogue scales in rating subjective feelings , 1974 .

[63]  Jill X. O'Reilly,et al.  Making predictions in a changing world—inference, uncertainty, and learning , 2013, Front. Neurosci..

[64]  G. Rainer,et al.  Cognitive neuroscience: Neural mechanisms for detecting and remembering novel events , 2003, Nature Reviews Neuroscience.

[65]  G. Mangun Neural mechanisms of visual selective attention. , 1995, Psychophysiology.

[66]  Markus Ullsperger,et al.  Adaptive Coding of Action Values in the Human Rostral Cingulate Zone , 2009, The Journal of Neuroscience.

[67]  Ruth Seurinck,et al.  The influence of the noradrenergic system on optimal control of neural plasticity , 2013, Front. Behav. Neurosci..

[68]  Jonathan D. Cohen,et al.  Decision making, the P3, and the locus coeruleus-norepinephrine system. , 2005, Psychological bulletin.

[69]  M. Hasselmo Neuromodulation and cortical function: modeling the physiological basis of behavior , 1995, Behavioural Brain Research.

[70]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[71]  Mark S. Gilzenrat,et al.  Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function , 2010, Cognitive, affective & behavioral neuroscience.

[72]  M Hansenne [The p300 cognitive event-related potential. I. Theoretical and psychobiologic perspectives]. , 2000, Neurophysiologie clinique = Clinical neurophysiology.

[73]  J D Cohen,et al.  A network model of catecholamine effects: gain, signal-to-noise ratio, and behavior. , 1990, Science.

[74]  N. Bolger,et al.  Brain Mediators of Predictive Cue Effects on Perceived Pain , 2010, The Journal of Neuroscience.

[75]  T. Robbins,et al.  Lesions of the dorsal noradrenergic bundle impair attentional set‐shifting in the rat , 2007, The European journal of neuroscience.

[76]  C. Berridge,et al.  The locus coeruleus–noradrenergic system: modulation of behavioral state and state-dependent cognitive processes , 2003, Brain Research Reviews.

[77]  M. Frank,et al.  Conflict acts as an implicit cost in reinforcement learning , 2014, Nature Communications.

[78]  Berrin Maraşligil,et al.  İnsanlarda Yenilik N2 Yanıtı Hedef Uyaranların Zamansal Sınıflamasını Yansıtır , 2011 .

[79]  J. Polich Updating P300: An integrative theory of P3a and P3b , 2007, Clinical Neurophysiology.

[80]  J. Ford,et al.  The relationship between P300 amplitude and regional gray matter volumes depends upon the attentional system engaged. , 1994, Electroencephalography and clinical neurophysiology.

[81]  Peter Bossaerts,et al.  The Neural Representation of Unexpected Uncertainty during Value-Based Decision Making , 2013, Neuron.

[82]  D. Morilak,et al.  Chronic Treatment with Desipramine Improves Cognitive Performance of Rats in an Attentional Set-Shifting Test , 2007, Neuropsychopharmacology.

[83]  B. Waterhouse,et al.  Retrograde double‐labeling study of common afferent projections to the dorsal raphe and the nuclear core of the locus coeruleus in the rat , 2005, The Journal of comparative neurology.

[84]  H. Eichenbaum,et al.  Noradrenergic, but not cholinergic, deafferentation of prefrontal cortex impairs attentional set-shifting , 2008, Neuroscience.

[85]  Angela J. Yu,et al.  Uncertainty, Neuromodulation, and Attention , 2005, Neuron.

[86]  John-Michael Sauer,et al.  Clinical Pharmacokinetics of Atomoxetine , 2005, Clinical pharmacokinetics.

[87]  C. Swanson,et al.  Effect of the attention deficit/hyperactivity disorder drug atomoxetine on extracellular concentrations of norepinephrine and dopamine in several brain regions of the rat , 2006, Neuropharmacology.

[88]  M. D’Esposito,et al.  Inverted-U–Shaped Dopamine Actions on Human Working Memory and Cognitive Control , 2011, Biological Psychiatry.

[89]  J. Gold,et al.  Relationships between Pupil Diameter and Neuronal Activity in the Locus Coeruleus, Colliculi, and Cingulate Cortex , 2016, Neuron.

[90]  Paul J. Harrison,et al.  Catechol-o-Methyltransferase, Cognition, and Psychosis: Val158Met and Beyond , 2006, Biological Psychiatry.

[91]  Martin A. Lindquist,et al.  Brain mediators of cardiovascular responses to social threat, Part II: Prefrontal-subcortical pathways and relationship with anxiety , 2009, NeuroImage.

[92]  E. Seu,et al.  Inhibition of the norepinephrine transporter improves behavioral flexibility in rats and monkeys , 2008, Psychopharmacology.

[93]  Robert C. Wilson,et al.  Rational regulation of learning dynamics by pupil–linked arousal systems , 2012, Nature Neuroscience.

[94]  R. O’Connell,et al.  Pupil diameter covaries with BOLD activity in human locus coeruleus , 2014, Human brain mapping.

[95]  William D. Penny,et al.  The Role of Dopamine in Temporal Uncertainty , 2016, Journal of Cognitive Neuroscience.

[96]  G. Flore,et al.  On the origin of cortical dopamine: is it a co-transmitter in noradrenergic neurons? , 2006, Current neuropharmacology.

[97]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[98]  N. Yeung,et al.  Neural basis of motivational and cognitive control , 2011 .

[99]  Raymond J. Dolan,et al.  Dopamine, Affordance and Active Inference , 2012, PLoS Comput. Biol..

[100]  M. Frank,et al.  Striatal Dopamine Predicts Outcome-Specific Reversal Learning and Its Sensitivity to Dopaminergic Drug Administration , 2009, The Journal of Neuroscience.

[101]  Robert C. Wilson,et al.  An Approximately Bayesian Delta-Rule Model Explains the Dynamics of Belief Updating in a Changing Environment , 2010, The Journal of Neuroscience.