The effect of atomoxetine on random and directed exploration in humans

The adaptive regulation of the trade-off between pursuing a known reward (exploitation) and sampling lesser-known options in search of something better (exploration) is critical for optimal performance. Theory and recent empirical work suggest that humans use at least two strategies for solving this dilemma: a directed strategy in which choices are explicitly biased toward information seeking, and a random strategy in which decision noise leads to exploration by chance. Here we examined the hypothesis that random exploration is governed by the neuromodulatory locus coeruleus-norepinephrine system. We administered atomoxetine, a norepinephrine transporter blocker that increases extracellular levels of norepinephrine throughout the cortex, to 22 healthy human participants in a double-blind crossover design. We examined the effect of treatment on performance in a gambling task designed to produce distinct measures of directed exploration and random exploration. In line with our hypothesis we found an effect of atomoxetine on random, but not directed exploration. However, contrary to expectation, atomoxetine reduced rather than increased random exploration. We offer three potential explanations of our findings, involving the non-linear relationship between tonic NE and cognitive performance, the interaction of atomoxetine with other neuromodulators, and the possibility that atomoxetine affected phasic norepinephrine activity more so than tonic norepinephrine activity.

[1]  W. R. Thompson ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLES , 1933 .

[2]  J. Gittins Bandit processes and dynamic allocation indices , 1979 .

[3]  Rakesh K. Sarin,et al.  Modeling Ambiguity in Decisions Under Uncertainty , 1988 .

[4]  S. Foote,et al.  Corticotropin-releasing hormone increases tonic but not sensory-evoked activity of noradrenergic locus coeruleus neurons in unanesthetized rats , 1988, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[5]  John S. Bridle,et al.  Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters , 1989, NIPS.

[6]  J D Cohen,et al.  A network model of catecholamine effects: gain, signal-to-noise ratio, and behavior. , 1990, Science.

[7]  Colin Camerer,et al.  Recent developments in modeling preferences: Uncertainty and ambiguity , 1992 .

[8]  G. Aston-Jones,et al.  Locus coeruleus activity in monkey: Phasic and tonic changes are associated with altered vigilance , 1994, Brain Research Bulletin.

[9]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[10]  J. Cohen,et al.  The role of locus coeruleus in the regulation of cognitive performance. , 1999, Science.

[11]  K. Perry,et al.  Atomoxetine Increases Extracellular Levels of Norepinephrine and Dopamine in Prefrontal Cortex of Rat: A Potential Mechanism for Efficacy in Attention Deficit/Hyperactivity Disorder , 2002, Neuropsychopharmacology.

[12]  Jonathan D. Cohen,et al.  Simplified dynamics in a model of noradrenergic modulation of cognitive performance , 2002, Neural Networks.

[13]  G. Badger,et al.  Comparison of the subjective, physiological, and psychomotor effects of atomoxetine and methylphenidate in light drug users. , 2002, Drug and alcohol dependence.

[14]  C. Berridge,et al.  The locus coeruleus–noradrenergic system: modulation of behavioral state and state-dependent cognitive processes , 2003, Brain Research Reviews.

[15]  Peter Auer,et al.  Finite-time Analysis of the Multiarmed Bandit Problem , 2002, Machine Learning.

[16]  Silvio Garattini,et al.  Role of presynaptic α2-adrenoceptors in antidepressant action: recent findings from microdialysis studies , 2004, Progress in Neuro-Psychopharmacology and Biological Psychiatry.

[17]  Jonathan D. Cohen,et al.  An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. , 2005, Annual review of neuroscience.

[18]  Angela J. Yu,et al.  Uncertainty, Neuromodulation, and Attention , 2005, Neuron.

[19]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[20]  John-Michael Sauer,et al.  Clinical Pharmacokinetics of Atomoxetine , 2005, Clinical pharmacokinetics.

[21]  C. Swanson,et al.  Effect of the attention deficit/hyperactivity disorder drug atomoxetine on extracellular concentrations of norepinephrine and dopamine in several brain regions of the rat , 2006, Neuropharmacology.

[22]  Angela J. Yu,et al.  Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[23]  B. Roth,et al.  Evolving the lock to fit the key to create a family of G protein-coupled receptors potently activated by an inert ligand , 2007, Proceedings of the National Academy of Sciences.

[24]  T. Robbins,et al.  Atomoxetine increases salivary cortisol in healthy volunteers , 2007, Journal of psychopharmacology.

[25]  W. Gerstner,et al.  Stress, genotype and norepinephrine in the prediction of mouse behavior using reinforcement learning , 2009, Nature Neuroscience.

[26]  M. Nicolelis,et al.  Remote Control of Neuronal Activity in Transgenic Mice Expressing Evolved G Protein-Coupled Receptors , 2009, Neuron.

[27]  M. Frank,et al.  Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. , 2009, Nature neuroscience.

[28]  Mark S. Gilzenrat,et al.  Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function , 2010, Cognitive, affective & behavioral neuroscience.

[29]  Toshio Matsuda,et al.  Effects of acute and chronic administration of atomoxetine and methylphenidate on extracellular levels of noradrenaline, dopamine and serotonin in the prefrontal cortex and striatum of mice , 2010, Journal of neurochemistry.

[30]  S. Cooper,et al.  Remote Control , 2002, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[31]  Eric-Jan Wagenmakers,et al.  The Role of the Noradrenergic System in the Exploration–Exploitation Trade-Off: A Psychopharmacological Study , 2010, Front. Hum. Neurosci..

[32]  Berrin Maraşligil,et al.  İnsanlarda Yenilik N2 Yanıtı Hedef Uyaranların Zamansal Sınıflamasını Yansıtır , 2011 .

[33]  S. Nieuwenhuis,et al.  The anatomical and functional relationship between the P3 and autonomic components of the orienting response. , 2011, Psychophysiology.

[34]  Sander Nieuwenhuis,et al.  Pupil Diameter Predicts Changes in the Exploration–Exploitation Trade-off: Evidence for the Adaptive Gain Theory , 2011, Journal of Cognitive Neuroscience.

[35]  M. Frank,et al.  Deficits in Positive Reinforcement Learning and Uncertainty-Driven Exploration Are Associated with Distinct Aspects of Negative Symptoms in Schizophrenia , 2011, Biological Psychiatry.

[36]  Sébastien Bubeck,et al.  Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems , 2012, Found. Trends Mach. Learn..

[37]  Jonathan D. Cohen,et al.  The effects of neural gain on attention and learning , 2013, Nature Neuroscience.

[38]  G. Aston-Jones,et al.  Atomoxetine modulates spontaneous and sensory-evoked discharge of locus coeruleus noradrenergic neurons , 2013, Neuropharmacology.

[39]  J. Kruschke Bayesian estimation supersedes the t test. , 2013, Journal of experimental psychology. General.

[40]  C. Correll,et al.  Efficacy and safety of atomoxetine in children and adolescents with attention-deficit/hyperactivity disorder: results from a comprehensive meta-analysis and metaregression. , 2014, Journal of the American Academy of Child and Adolescent Psychiatry.

[41]  Jonathan D. Cohen,et al.  Humans use directed and random exploration to solve the explore-exploit dilemma. , 2014, Journal of experimental psychology. General.

[42]  K. Branson,et al.  Behavioral Variability through Stochastic Choice and Its Gating by Anterior Cingulate Cortex , 2014, Cell.

[43]  Joachim Vandekerckhove,et al.  Pupil-Linked Arousal Determines Variability in Perceptual Decision Making , 2014, PLoS Comput. Biol..

[44]  Thomas V. Wiecki,et al.  Eye tracking and pupillometry are indicators of dissociable latent decision processes. , 2014, Journal of experimental psychology. General.

[45]  R. O’Connell,et al.  Pupil diameter covaries with BOLD activity in human locus coeruleus , 2014, Human brain mapping.

[46]  M. Lee,et al.  Bayesian Cognitive Modeling: A Practical Course , 2014 .

[47]  Michael J Frank,et al.  Dopamine, Locus of Control, and the Exploration-Exploitation Tradeoff , 2015, Neuropsychopharmacology.

[48]  Yael Niv,et al.  Do You See the Forest or the Tree? Neural Gain and Breadth Versus Focus in Perceptual Processing , 2016, Psychological science.

[49]  J. Gold,et al.  Relationships between Pupil Diameter and Neuronal Activity in the Locus Coeruleus, Colliculi, and Cingulate Cortex , 2016, Neuron.

[50]  J. Cohen,et al.  Catecholamine-Mediated Increases in Gain Enhance the Precision of Cortical Representations , 2016, The Journal of Neuroscience.

[51]  Robert C. Wilson,et al.  Charting the Expansion of Strategic Exploratory Behavior During Adolescence , 2017, Journal of experimental psychology. General.

[52]  Robert C. Wilson,et al.  A causal role for right frontopolar cortex in directed, but not random, exploration , 2016, bioRxiv.