A robot model of the basal ganglia: Behavior and intrinsic processing

The existence of multiple parallel loops connecting sensorimotor systems to the basal ganglia has given rise to proposals that these nuclei serve as a selection mechanism resolving competitions between the alternative actions available in a given context. A strong test of this hypothesis is to require a computational model of the basal ganglia to generate integrated selection sequences in an autonomous agent, we therefore describe a robot architecture into which such a model is embedded, and require it to control action selection in a robotic task inspired by animal observations. Our results demonstrate effective action selection by the embedded model under a wide range of sensory and motivational conditions. When confronted with multiple, high salience alternatives, the robot also exhibits forms of behavioral disintegration that show similarities to animal behavior in conflict situations. The model is shown to cast light on recent neurobiological findings concerning behavioral switching and sequencing.

[1]  Lui Sha,et al.  Priority Inheritance Protocols: An Approach to Real-Time Synchronization , 1990, IEEE Trans. Computers.

[2]  G. E. Alexander,et al.  Functional architecture of basal ganglia circuits: neural substrates of parallel processing , 1990, Trends in Neurosciences.

[3]  C. Barnard Problems of Animal Behaviour, David McFarland. Longman, Harlow, Essex (1989), vii, + 158. Price £12.95 , 1990 .

[4]  George V Rebec,et al.  Behavior-related changes in the activity of substantia nigra pars reticulata neurons in freely moving rats , 1999, Brain Research.

[5]  Joel L. Davis,et al.  A Model of How the Basal Ganglia Generate and Use Neural Signals That Predict Reinforcement , 1994 .

[6]  W. Schultz,et al.  Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[7]  J. W. Aldridge,et al.  Coding of Serial Order by Neostriatal Neurons: A “Natural Action” Approach to Movement Sequence , 1998, The Journal of Neuroscience.

[8]  W. T. Thach,et al.  Basal ganglia motor control. II. Late pallidal timing relative to movement onset and inconsistent pallidal coding of movement parameters. , 1991, Journal of neurophysiology.

[9]  T. Tsumori,et al.  Organization of the nigro-tecto-bulbar pathway to the parvicellular reticular formation: a light- and electron-microscopic study in the rat , 1997, Experimental Brain Research.

[10]  M. A. Basso,et al.  Neuronal Activity in Substantia Nigra Pars Reticulata during Target Selection , 2002, The Journal of Neuroscience.

[11]  Peter Ford Dominey,et al.  A cortico-subcortical model for generation of spatially accurate sequential saccades. , 1992, Cerebral cortex.

[12]  G. Percheron,et al.  The Basal Ganglia IV , 1994, Advances in Behavioral Biology.

[13]  A. Guillot,et al.  A basal ganglia inspired model of action selection evaluated in a robotic survival task. , 2003, Journal of integrative neuroscience.

[14]  J. Wickens,et al.  Computational models of the basal ganglia: from robots to membranes , 2004, Trends in Neurosciences.

[15]  C. Marsden,et al.  The functions of the basal ganglia and the paradox of stereotaxic surgery in Parkinson's disease. , 1994, Brain : a journal of neurology.

[16]  D. Massaro Some criticisms of connectionist models of human performance , 1988 .

[17]  M. West,et al.  Loss of Lever Press-Related Firing of Rat Striatal Forelimb Neurons after Repeated Sessions in a Lever Pressing Task , 1997, The Journal of Neuroscience.

[18]  Pattie Maes,et al.  Modeling Adaptive Autonomous Agents , 1993, Artificial Life.

[19]  J. Penney,et al.  The functional anatomy of basal ganglia disorders , 1989, Trends in Neurosciences.

[20]  M. Zigmond,et al.  Influence of dopamine on GABA release in striatum: evidence for D1–D2 interactions and non-synaptic influences , 1997, Neuroscience.

[21]  John G. Taylor,et al.  Analysis of recurrent cortico-basal ganglia-thalamic loops for working memory , 2000, Biological Cybernetics.

[22]  T. Robbins,et al.  The Role of the Striatum in the Mental Chronometry of Action: A Theoretical Review , 1990, Reviews in the neurosciences.

[23]  P. Redgrave,et al.  The basal ganglia: a vertebrate solution to the selection problem? , 1999, Neuroscience.

[24]  R. Hinde,et al.  The Conflict Between Drives in the Courtship and Copulation of the Chaffinch , 1953 .

[25]  Michael A. Arbib,et al.  The handbook of brain theory and neural networks , 1995, A Bradford book.

[26]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[27]  Mark D. Humphries,et al.  A pulsed neural network model of bursting in the basal ganglia , 2001, Neural Networks.

[28]  S. Wiener,et al.  Neurons in hippocampal afferent zones of rat striatum parse routes into multi‐pace segments during maze navigation , 2004, The European journal of neuroscience.

[29]  A P Georgopoulos,et al.  Role of basal ganglia in limb movements. , 1984, Human neurobiology.

[30]  W. Meck,et al.  Neuropsychological mechanisms of interval timing behavior. , 2000, BioEssays : news and reviews in molecular, cellular and developmental biology.

[31]  H. Evans The Study of Instinct , 1952 .

[32]  K. Lorenz,et al.  Der Kumpan in der Umwelt des Vogels , 1935, Journal für Ornithologie.

[33]  Stephen Grossberg,et al.  How laminar frontal cortex and basal ganglia circuits interact to control planned and reactive saccades , 2004, Neural Networks.

[34]  R B MALMO,et al.  Activation: a neuropsychological dimension. , 1959, Rassegna giuliana di medicina.

[35]  D. Bindra,et al.  AN INTERPRETATION OF THE ‘DISPLACEMENT’ PHENOMENON* , 1959 .

[36]  P. Colgan,et al.  Animal Motivation , 1989, Chapman and Hall Animal Behaviour Series.

[37]  Eytan Ruppin,et al.  Actor-critic models of the basal ganglia: new anatomical and computational perspectives , 2002, Neural Networks.

[38]  Peter Redgrave,et al.  Basal Ganglia , 2020, Encyclopedia of Autism Spectrum Disorders.

[39]  W. Schultz Dopamine neurons and their role in reward mechanisms , 1997, Current Opinion in Neurobiology.

[40]  Tomoki Fukai,et al.  Sequence generation in arbitrary temporal patterns from theta-nested gamma oscillations: a model of the basal ganglia-thalamo-cortical loops , 1999, Neural Networks.

[41]  D. Boussaoud,et al.  Role of the primate striatum in attention and sensorimotor processes: comparison with premotor cortex , 1995, Neuroreport.

[42]  Y. Smith,et al.  Efferent connections of the internal globus pallidus in the squirrel monkey: II. topography and synaptic organization of pallidal efferents to the pedunculopontine nucleus , 1997, The Journal of comparative neurology.

[43]  P. Redgrave,et al.  Testing computational hypotheses of brain systems function: a case study with the basal ganglia , 2004, Network.

[44]  Stephen M. Rao,et al.  Neural basis for impaired time reproduction in Parkinson's disease: An fMRI study , 2003, Journal of the International Neuropsychological Society.

[45]  S. Wiener,et al.  Position sensitivity in phasically discharging nucleus accumbens neurons of rats alternating between tasks requiring complementary types of spatial cues , 2001, Neuroscience.

[46]  W. Meck Selective adjustment of the speed of internal clock and memory processes. , 1983, Journal of experimental psychology. Animal behavior processes.

[47]  F. Toates The interaction of cognitive and stimulus–response processes in the control of behaviour , 1997, Neuroscience & Biobehavioral Reviews.

[48]  Christopher G. Langton,et al.  Artificial Life , 2019, Philosophical Posthumanism.

[49]  W. Schultz Activity of pars reticulata neurons of monkey substantia nigra in relation to motor, sensory, and complex events. , 1986, Journal of neurophysiology.

[50]  D. Ulrich Differential arithmetic of shunting inhibition for voltage and spike rate in neocortical pyramidal cells , 2003, The European journal of neuroscience.

[51]  D. McFarland,et al.  Intelligent behavior in animals and robots , 1993 .

[52]  Y. Smith,et al.  Microcircuitry of the direct and indirect pathways of the basal ganglia. , 1998, Neuroscience.

[53]  G. Arbuthnott,et al.  Computational models of the basal ganglia , 2000, Movement disorders : official journal of the Movement Disorder Society.

[54]  J. Houk,et al.  Model of cortical-basal ganglionic processing: encoding the serial order of sensory events. , 1998, Journal of neurophysiology.

[55]  Florentin Wörgötter,et al.  Temporal Sequence Learning, Prediction, and Control: A Review of Different Models and Their Relation to Biological Mechanisms , 2005, Neural Computation.

[56]  Jonathan D. Cohen,et al.  Computational roles for dopamine in behavioural control , 2004, Nature.

[57]  P. Klopfer,et al.  Perspectives in Ethology , 1973, Springer US.

[58]  A. Parent,et al.  Functional anatomy of the basal ganglia. II. The place of subthalamic nucleus and external pallidium in basal ganglia circuitry , 1995, Brain Research Reviews.

[59]  J. W. Aldridge,et al.  Substantia nigra pars reticulata neurons code initiation of a serial pattern: implications for natural action sequences and sequential disorders , 2002, The European journal of neuroscience.

[60]  J. T. Murphy,et al.  The role of the basal ganglia in controlling a movement initiated by a visually presented cue , 1980, Brain Research.

[61]  R. Hinde,et al.  Animal Behavior: A Synthesis of Ethology and Comparative Psychology , 1967 .

[62]  M. Deschenes,et al.  Corticostriatal projections from layer V cells in rat are collaterals of long-range corticofugal axons , 1996, Brain Research.

[63]  Peter Redgrave,et al.  A computational model of action selection in the basal ganglia. II. Analysis and simulation of behaviour , 2001, Biological Cybernetics.

[64]  H. Bergman,et al.  Information processing, dimensionality reduction and reinforcement learning in the basal ganglia , 2003, Progress in Neurobiology.

[65]  W. Smeets,et al.  Anatomical Substrate of Amphibian Basal Ganglia Involvement in Visuomotor Behaviour , 1997, The European journal of neuroscience.

[66]  A. Cools Role of the neostriatal dopaminergic activity in sequencing and selecting behavioural strategies: Facilitation of processes involved in selecting the best strategy in a stressful situation , 1980, Behavioural Brain Research.

[67]  J. Yelnik Functional anatomy of the basal ganglia , 2002, Movement disorders : official journal of the Movement Disorder Society.

[68]  Peter Redgrave,et al.  Layered Control Architectures in Robots and Vertebrates , 1999, Adapt. Behav..

[69]  D. Denny-Brown,et al.  The role of the basal ganglia in the initiation of movement. , 1976, Research publications - Association for Research in Nervous and Mental Disease.

[70]  Charles J. Wilson,et al.  Chapter II The basal ganglia , 1996 .

[71]  José Luis Contreras-Vidal,et al.  A neural model of basal ganglia-thalamocortical relations in normal and parkinsonian movement , 1995, Biological Cybernetics.

[72]  J. Mink THE BASAL GANGLIA: FOCUSED SELECTION AND INHIBITION OF COMPETING MOTOR PROGRAMS , 1996, Progress in Neurobiology.

[73]  A. Grace Phasic versus tonic dopamine release and the modulation of dopamine system responsivity: A hypothesis for the etiology of schizophrenia , 1991, Neuroscience.

[74]  T. Poggio,et al.  Nonlinear interactions in a dendritic tree: localization, timing, and role in information processing. , 1983, Proceedings of the National Academy of Sciences of the United States of America.

[75]  C. Gerfen,et al.  D1 and D2 dopamine receptor-regulated gene expression of striatonigral and striatopallidal neurons. , 1990, Science.

[76]  Catalin V Buhusi,et al.  Differential effects of methamphetamine and haloperidol on the control of an internal clock. , 2002, Behavioral neuroscience.

[77]  Peter Redgrave,et al.  A computational model of action selection in the basal ganglia. I. A new functional anatomy , 2001, Biological Cybernetics.

[78]  M. Kimura Role of basal ganglia in behavioral learning , 1995, Neuroscience Research.

[79]  K. Clark,et al.  The role of the subthalamic nucleus in the response of globus pallidus neurons to stimulation of the prelimbic and agranular frontal cortices in rats , 2004, Experimental Brain Research.

[80]  P. Redgrave,et al.  Is the short-latency dopamine response too short to signal reward error? , 1999, Trends in Neurosciences.

[81]  G. E. Reeves,et al.  What Really Happened on Mars , 1998 .

[82]  C. Wilson,et al.  Spontaneous firing patterns and axonal projections of single corticostriatal neurons in the rat medial agranular cortex. , 1994, Journal of neurophysiology.

[83]  B. Biguer,et al.  Activity of neurons in the cat substantia nigra pars reticulata during drinking , 2004, Experimental Brain Research.

[84]  G. Pagnoni,et al.  Human Striatal Response to Salient Nonrewarding Stimuli , 2003, The Journal of Neuroscience.

[85]  O. Hikosaka Role of Basal Ganglia in Control of Innate Movements, Learned Behavior and Cognition—A Hypothesis , 1994 .

[86]  Kenji Doya,et al.  Introduction for 2002 Special Issue: Computational Models of Neuromodulation , 2002, Neural Networks.

[87]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[88]  R. Yerkes,et al.  The relation of strength of stimulus to rapidity of habit‐formation , 1908 .

[89]  William Rowan,et al.  The Study of Instinct , 1953 .

[90]  Olaf Sporns,et al.  Neuromodulation and plasticity in an autonomous robot , 2002, Neural Networks.

[91]  A. Graybiel The Basal Ganglia and Chunking of Action Repertoires , 1998, Neurobiology of Learning and Memory.

[92]  W. Meck,et al.  Dissecting the Brain's Internal Clock: How Frontal–Striatal Circuitry Keeps Time and Shifts Attention , 2002, Brain and Cognition.

[93]  A. Redish,et al.  Neuronal activity in the rodent dorsal striatum in sequential navigation: separation of spatial and reward responses on the multiple T task. , 2004, Journal of neurophysiology.

[94]  J. Joseph,et al.  Activity in the caudate nucleus of monkey during spatial sequencing. , 1995, Journal of neurophysiology.

[95]  J. Hollerman,et al.  Involvement of basal ganglia and orbitofrontal cortex in goal-directed behavior. , 2000, Progress in brain research.

[96]  K. Berridge,et al.  Implementation of Action Sequences by a Neostriatal Site: A Lesion Mapping Study of Grooming Syntax , 1996, The Journal of Neuroscience.

[97]  Stewart W. Wilson,et al.  From Animals to Animats 5. Proceedings of the Fifth International Conference on Simulation of Adaptive Behavior , 1997 .

[98]  S. Pinker,et al.  On language and connectionism: Analysis of a parallel distributed processing model of language acquisition , 1988, Cognition.

[99]  John C. Fentress,et al.  Specific and Nonspecific Factors in the Causation of Behavior , 1973 .

[100]  Philip N. Lehner,et al.  Handbook of ethological methods , 1979 .

[101]  S P Wise,et al.  Distributed modular architectures linking basal ganglia, cerebellum, and cerebral cortex: their role in planning and controlling action. , 1995, Cerebral cortex.

[102]  Jack Ganssle,et al.  Embedded Systems Dictionary , 2003 .

[103]  G. Rebec,et al.  Behavior-related modulation of substantia nigra pars reticulata neurons in rats performing a conditioned reinforcement task , 2002, Neuroscience.

[104]  Humphries The basal ganglia and action selection : a computational study at multiple levels of description. , 2002 .

[105]  E Covey,et al.  A neuroethological theory of the operation of the inferior colliculus. , 1996, Brain, behavior and evolution.

[106]  B. Bioulac,et al.  Responses of substantia nigra pars reticulata neurons to intrastriatal D1 and D2 dopaminergic agonist injections in the rat , 1996, Neuroscience Letters.

[107]  P. Redgrave,et al.  An embodied model of action selection mechanisms in the vertebrate brain , 2000 .

[108]  Kelly A. Allers,et al.  Pre- and postsynaptic aspects of dopamine-mediated transmission , 2000, Trends in Neurosciences.

[109]  P. Dayan,et al.  Reward, Motivation, and Reinforcement Learning , 2002, Neuron.

[110]  M. Deschenes,et al.  Corticothalamic projections from layer V cells in rat are collaterals of long-range corticofugal axons , 1994, Brain Research.

[111]  M D Humphries,et al.  The role of intra-thalamic and thalamocortical circuits in action selection , 2002, Network.

[112]  K. Lorenz,et al.  Der Kumpan in der Umwelt des Vogels , 1935, Journal für Ornithologie.

[113]  R M Church,et al.  Scalar Timing in Memory , 1984, Annals of the New York Academy of Sciences.

[114]  Thomas A. Sebeok,et al.  How Animals Communicate , 1979 .

[115]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[116]  Alasdair Houston,et al.  A positive feedback model for switching between two activities , 1985, Animal Behaviour.

[117]  Michael J. Frank,et al.  Interactions between frontal cortex and basal ganglia in working memory: A computational model , 2001, Cognitive, affective & behavioral neuroscience.

[118]  Joanna Bryson,et al.  Cross-paradigm analysis of autonomous agent architecture , 2000, J. Exp. Theor. Artif. Intell..

[119]  J. Deniau,et al.  Spatio-temporal organization of a branched tecto-spinal/ tecto-diencephalic neuronal system , 1984, Neuroscience.

[120]  Eugene M. Izhikevich,et al.  Simple model of spiking neurons , 2003, IEEE Trans. Neural Networks.

[121]  Dieter Jaeger,et al.  Neuronal activity in the striatum and pallidum of primates related to the execution of externally cued reaching movements , 1995, Brain Research.

[122]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[123]  J. Ewert Neuroethology of releasing mechanisms: Prey-catching in toads , 1987, Behavioral and Brain Sciences.

[124]  D. Joel,et al.  The organization of the basal ganglia-thalamocortical circuits: Open interconnected rather than closed segregated , 1994, Neuroscience.

[125]  Garrett E. Alexander Basal ganglia , 1998 .

[126]  J. Wickens Basal ganglia: structure and computations. , 1997 .

[127]  Alan C. Secord,et al.  Animal Behaviour–A Synthesis of Ethology and Comparative Psychology. , 1967 .

[128]  Joel L. Davis,et al.  Adaptive Critics and the Basal Ganglia , 1995 .

[129]  Kevin N. Gurney,et al.  A novel parameter optimisation technique for compartmental models applied to a model of a striatal medium spiny neuron , 2004, Neurocomputing.

[130]  G. Ermentrout,et al.  Analysis of neural excitability and oscillations , 1989 .

[131]  P Redgrave,et al.  Superior colliculus projections to midline and intralaminar thalamic nuclei of the rat , 2001, The Journal of comparative neurology.

[132]  G. Hoyle The scope of neuroethology , 1984, Behavioral and Brain Sciences.

[133]  Charles J. Wilson,et al.  The origins of two-state spontaneous membrane potential fluctuations of neostriatal spiny neurons , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[134]  P. Glimcher,et al.  Quantitative analysis of substantia nigra pars reticulata activity during a visually guided saccade task. , 1999, Journal of neurophysiology.

[135]  Peter Ford Dominey,et al.  Encoding behavioral context in recurrent networks of the fronto-striatal system: a simulation study. , 1997, Brain research. Cognitive brain research.

[136]  Mark D. Humphries,et al.  The Interaction of Recurrent Axon Collateral Networks in the Basal Ganglia , 2003, ICANN.

[137]  S. Blomfield Arithmetical operations performed by nerve cells. , 1974, Brain research.

[138]  R. Wurtz,et al.  The Neurobiology of Saccadic Eye Movements , 1989 .

[139]  J. Ewert,et al.  Neural modulation of visuomotor functions underlying prey-catching behaviour in anurans: perception, attention, motor performance, learning. , 2001, Comparative biochemistry and physiology. Part A, Molecular & integrative physiology.