A neurorobotic platform to test the influence of neuromodulatory signaling on anxious and curious behavior

The vertebrate neuromodulatory systems are critical for appropriate value-laden responses to environmental challenges. Whereas changes in the overall level of dopamine (DA) have an effect on the organism's reward or curiosity-seeking behavior, changes in the level of serotonin (5-HT) can affect its level of anxiety or harm aversion. Moreover, top-down signals from frontal cortex can exert cognitive control on these neuromodulatory systems. The cholinergic (ACh) and noradrenergic (NE) systems affect the ability to filter out noise and irrelevant events. We introduce a neural network for action selection that is based on these principles of neuromodulatory systems. The algorithm tested the hypothesis that high levels of serotonin lead to withdrawn behavior by suppressing DA action and that high levels of DA or low levels of 5-HT lead to curious, exploratory behavior. Furthermore, the algorithm tested the idea that top-down signals from the frontal cortex to neuromodulatory areas are critical for an organism to cope with both stressful and novel events. The neural network was implemented on an autonomous robot and tested in an open-field paradigm. The open-field test is often used to test for models anxiety or exploratory behavior in the rodent and allows for qualitative comparisons with the neurorobot's behavior. The present neurorobotic experiments can lead to a better understanding of how neuromodulatory signaling affects the balance between anxious and curious behavior. Therefore, this experimental paradigm may also be informative in exploring a wide range of neurological diseases such as anxiety, autism, attention deficit disorders, and obsessive-compulsive disorders.

[1]  M. Frank,et al.  Dopaminergic Genes Predict Individual Differences in Susceptibility to Confirmation Bias , 2011, The Journal of Neuroscience.

[2]  M. Shapiro,et al.  Reward Stability Determines the Contribution of Orbitofrontal Cortex to Adaptive Behavior , 2012, The Journal of Neuroscience.

[3]  Jonathan D. Cohen,et al.  An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. , 2005, Annual review of neuroscience.

[4]  M. Walton,et al.  Differential involvement of serotonin and dopamine systems in cost-benefit decisions about delay or effort , 2005, Psychopharmacology.

[5]  Olaf Sporns,et al.  Neuromodulation and plasticity in an autonomous robot , 2002, Neural Networks.

[6]  Jeffrey L. Krichmar,et al.  as a Robot Controller Neuromodulation as a Robot Controller A Brain-Inspired Strategy for Controlling Autonomous Robots , 2009 .

[7]  E. Rolls.  The functions of the orbitofrontal cortex , 1999, Brain and Cognition.

[8]  Pamela A. Silver,et al.  Cortical 5-HT 2A Receptor Signaling Modulates Anxiety-Like Behaviors in Mice , 2006 .

[9]  Matthew D. Lieberman,et al.  Serotonin Modulates Behavioral Reactions to Unfairness , 2008, Science.

[10]  Anat Sakov,et al.  New replicable anxiety-related measures of wall vs center behavior of mice in the open field. , 2004, Journal of applied physiology.

[11]  Kenji Doya,et al.  Metalearning and neuromodulation , 2002, Neural Networks.

[12]  J. Feldon,et al.  Differential role of the medial and lateral prefrontal cortices in fear and anxiety. , 2000, Behavioral neuroscience.

[13]  M. Egan,et al.  Serotonin Transporter Genetic Variation and the Response of the Human Amygdala , 2002, Science.

[14]  D. Weinberger,et al.  Genes, dopamine and cortical signal-to-noise ratio in schizophrenia , 2004, Trends in Neurosciences.

[15]  A. Caspi,et al.  Influence of Life Stress on Depression: Moderation by a Polymorphism in the 5-HTT Gene , 2003, Science.

[16]  J. Deakin,et al.  Depression and antisocial personality disorder: two contrasting disorders of 5HT function. , 2003, Journal of neural transmission. Supplementum.

[17]  S. Maier,et al.  Medial prefrontal cortex determines how stressor controllability affects behavior and dorsal raphe nucleus , 2005, Nature Neuroscience.

[18]  Margit Burmeister,et al.  Serotonin transporter gene, stress and raphe–raphe interactions: a molecular mechanism of depression , 2012, Trends in Neurosciences.

[19]  Shohei Kato,et al.  A Dynamic Allocation Method of Basis Functions in Reinforcement Learning , 2004, Australian Conference on Artificial Intelligence.

[20]  Jeffrey L. Krichmar,et al.  The Neuromodulatory System: A Framework for Survival and Adaptive Behavior in a Challenging World , 2008, Adapt. Behav..

[21]  Jeffrey L. Krichmar,et al.  Simulation of How Neuromodulation Influences Cooperative Behavior , 2010, SAB.

[22]  Douglas A Nitz,et al.  Anterior cingulate neurons in the rat map anticipated effort and reward to their associated action sequences. , 2012, Journal of neurophysiology.

[23]  Saori C. Tanaka,et al.  Serotonin Differentially Regulates Short- and Long-Term Prediction of Rewards in the Ventral and Dorsal Striatum , 2007, PloS one.

[24]  M. Sarter,et al.  Modulators in concert for cognition: Modulator interactions in the prefrontal cortex , 2007, Progress in Neurobiology.

[25]  S. Maier,et al.  Role of the medial prefrontal cortex in coping and resilience , 2010, Brain Research.

[26]  Jeffrey L. Krichmar,et al.  Model of the interactions between neuromodulators and prefrontal cortex during a resource allocation task , 2012, 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL).

[27]  Sham M. Kakade,et al.  Opponent interactions between serotonin and dopamine , 2002, Neural Networks.

[28]  M. Millan,et al.  The neurobiology and control of anxious states , 2003, Progress in Neurobiology.

[29]  K. Berridge.  Motivation concepts in behavioral neuroscience , 2004, Physiology & Behavior.

[30]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[31]  M. Walton,et al.  Separate neural pathways process different decision costs , 2006, Nature Neuroscience.

[32]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[33]  Angela J. Yu,et al.  Uncertainty, Neuromodulation, and Attention , 2005, Neuron.

[34]  Ethan S. Bromberg-Martin,et al.  Dopamine in Motivational Control: Rewarding, Aversive, and Alerting , 2010, Neuron.

[35]  Eliza Congdon,et al.  Analysis of DRD4 and DAT polymorphisms and behavioral inhibition in healthy adults: Implications for impulsivity , 2008, American journal of medical genetics. Part B, Neuropsychiatric genetics : the official publication of the International Society of Psychiatric Genetics.

[36]  M. Hasselmo,et al.  High acetylcholine levels set circuit dynamics for attention and encoding and low acetylcholine levels set dynamics for consolidation. , 2004, Progress in brain research.

[37]  Andrew Holmes,et al.  Genetic variation in cortico-amygdala serotonin function and risk for stress-related disease , 2008, Neuroscience & Biobehavioral Reviews.

[38]  Arjan Blokland,et al.  Anxiogenic and depressive-like effects, but no cognitive deficits, after repeated moderate tryptophan depletion in the rat , 2002, Journal of psychopharmacology.

[39]  W. M. Keck,et al.  Machine Psychology : Autonomous Behavior , Perceptual Categorization and Conditioning in a Brain-based Device , 2002 .

[40]  Aude Billard,et al.  Reinforcement learning for imitating constrained reaching movements , 2007, Adv. Robotics.

[41]  R. A. Brooks,et al.  Intelligence without Representation , 1991, Artif. Intell..

[42]  Robert J. Carey,et al.  Cocaine effects on behavioral responding to a novel object placed in a familiar environment , 2008, Pharmacology Biochemistry and Behavior.

[43]  Shin Ishii,et al.  Reinforcement learning for a biped robot based on a CPG-actor-critic method , 2007, Neural Networks.

[44]  Anantha Shekhar,et al.  Differential effects of exposure to low-light or high-light open-field on anxiety-related behaviors; relationship to c-Fos expression in serotonergic and non-serotonergic neurons in the dorsal raphe nucleus , 2007, Brain Research Bulletin.

[45]  Angelo Arleo,et al.  Cognitive navigation based on nonuniform Gabor space sampling, unsupervised growing networks, and reinforcement learning , 2004, IEEE Transactions on Neural Networks.

[46]  S. Sara,et al.  Response to Novelty and its Rapid Habituation in Locus Coeruleus Neurons of the Freely Exploring Rat , 1995, The European journal of neuroscience.

[47]  J. John Mann,et al.  Cortical 5-HT2A Receptor Signaling Modulates Anxiety-Like Behaviors in Mice , 2006, Science.

[48]  M. Frank,et al.  Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. , 2006, Psychological review.

[49]  Peter Stone,et al.  Reinforcement Learning for RoboCup Soccer Keepaway , 2005, Adapt. Behav..

[50]  Maarten A. S. Boksem,et al.  Serotonin: Modulator of a drive to withdraw , 2009, Brain and Cognition.

[51]  Jeffrey L. Krichmar,et al.  Reciprocity and Retaliation in Social Games With Adaptive Agents , 2012, IEEE Transactions on Autonomous Mental Development.

[52]  Jeffrey L. Krichmar,et al.  Neuromodulation, attention and localization using a novel Android™ robotic platform , 2012, 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL).

[53]  Marcus R. Munafò,et al.  Association of the Dopamine D4 Receptor (DRD4) Gene and Approach-Related Personality Traits: Meta-Analysis and New Data , 2008, Biological Psychiatry.

[54]  P. Holland,et al.  Removal of Cholinergic Input to Rat Posterior Parietal Cortex Disrupts Incremental Processing of Conditioned Stimuli , 1998, The Journal of Neuroscience.

[55]  P. Dayan,et al.  Opponency Revisited: Competition and Cooperation Between Dopamine and Serotonin , 2010, Neuropsychopharmacology.

[56]  Timothy Edward John Behrens,et al.  Contrasting roles for cingulate and orbitofrontal cortex in decisions and social behaviour , 2007, Trends in Cognitive Sciences.

[57]  Jeffrey L. Krichmar,et al.  A biologically inspired action selection algorithm based on principles of neuromodulation , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[58]  Kenji Doya,et al.  The Cyber Rodent Project: Exploration of Adaptive Mechanisms for Self-Preservation and Self-Reproduction , 2005, Adapt. Behav..

[59]  Panos Roussos,et al.  Planning, decision-making and the COMT rs4818 polymorphism in healthy males , 2008, Neuropsychologia.

[60]  K. Doya.  Modulators of decision making , 2008, Nature Neuroscience.

[61]  O J Robinson,et al.  Mood state moderates the role of serotonin in cognitive biases , 2010, Journal of psychopharmacology.

[62]  M. Roesch,et al.  A new perspective on the role of the orbitofrontal cortex in adaptive behaviour , 2009, Nature Reviews Neuroscience.

[63]  Jeffrey L. Krichmar,et al.  Effect of neuromodulation on performance in game playing: A modeling study , 2010, 2010 IEEE 9th International Conference on Development and Learning.

[64]  Mark D. Humphries,et al.  A robot model of the basal ganglia: Behavior and intrinsic processing , 2006, Neural Networks.

[65]  Ahmad R. Hariri,et al.  Genetic Sensitivity to the Environment: The Case of the Serotonin Transporter Gene and Its Implications for Studying Complex Diseases and Traits , 2010 .

[66]  J. Costentin,et al.  Thigmotaxis as an index of anxiety in mice. Influence of dopaminergic transmissions , 1994, Behavioural Brain Research.

[67]  Saori C. Tanaka,et al.  Low-Serotonin Levels Increase Delayed Reward Discounting in Humans , 2008, The Journal of Neuroscience.

[68]  Yoav Benjamini,et al.  Freedom of movement and the stability of its unfolding in free exploration of mice , 2009, Proceedings of the National Academy of Sciences.

[69]  L H Parsons,et al.  Elevated anxiety and antidepressant-like responses in serotonin 5-HT1A receptor mutant mice. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[70]  Peter Ford Dominey,et al.  Robot Cognitive Control with a Neurophysiologically Inspired Reinforcement Learning Model , 2011, Front. Neurorobot..