Neuromodulated Synaptic Plasticity on the SpiNNaker Neuromorphic System

SpiNNaker is a digital neuromorphic architecture, designed specifically for the low power simulation of large-scale spiking neural networks at speeds close to biological real-time. Unlike other neuromorphic systems, SpiNNaker allows users to develop their own neuron and synapse models as well as specify arbitrary connectivity. As a result SpiNNaker has proved to be a powerful tool for studying different neuron models as well as synaptic plasticity—believed to be one of the main mechanisms behind learning and memory in the brain. A number of Spike-Timing-Dependent-Plasticity(STDP) rules have already been implemented on SpiNNaker and have been shown to be capable of solving various learning tasks in real-time. However, while STDP is an important biological theory of learning, it is a form of Hebbian or unsupervised learning and therefore does not explain behaviors that depend on feedback from the environment. Instead, learning rules based on neuromodulated STDP (three-factor learning rules) have been shown to be capable of solving reinforcement learning tasks in a biologically plausible manner. In this paper we demonstrate for the first time how a model of three-factor STDP, with the third-factor representing spikes from dopaminergic neurons, can be implemented on the SpiNNaker neuromorphic system. Using this learning rule we first show how reward and punishment signals can be delivered to a single synapse before going on to demonstrate it in a larger network which solves the credit assignment problem in a Pavlovian conditioning experiment. Because of its extra complexity, we find that our three-factor learning rule requires approximately 2× as much processing time as the existing SpiNNaker STDP learning rules. However, we show that it is still possible to run our Pavlovian conditioning model with up to 1 × 104 neurons in real-time, opening up new research opportunities for modeling behavioral learning on SpiNNaker.

[1]  Thomas Nowotny,et al.  Comparing Neuromorphic Solutions in Action: Implementing a Bio-Inspired Solution to a Benchmark Classification Task on Three Parallel-Computing Platforms , 2019 .

[2]  I. Pavlov,et al.  The work of the digestive glands : lectures , 1902 .

[3]  Giacomo Indiveri,et al.  The Cerebellum Chip: an Analog VLSI Implementation of a Cerebellar Model of Classical Conditioning , 2004, NIPS.

[4]  Razvan V. Florian,et al.  Reinforcement Learning Through Modulation of Spike-Timing-Dependent Synaptic Plasticity , 2007, Neural Computation.

[5]  Wulfram Gerstner,et al.  A neuronal learning rule for sub-millisecond temporal coding , 1996, Nature.

[6]  Johannes Schemmel,et al.  Demonstrating Hybrid Learning in a Flexible Neuromorphic Hardware System , 2016, IEEE Transactions on Biomedical Circuits and Systems.

[7]  Steve B. Furber,et al.  The SpiNNaker Project , 2014, Proceedings of the IEEE.

[8]  Patrice Y. Simard,et al.  High Performance Convolutional Neural Networks for Document Processing , 2006 .

[9]  Stephen B. Furber,et al.  Efficient modelling of spiking neural networks on a scalable chip multiprocessor , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[10]  J. Wickens,et al.  Timing is not Everything: Neuromodulation Opens the STDP Gate , 2010, Front. Syn. Neurosci..

[11]  E. Capaldi,et al.  The organization of behavior. , 1992, Journal of applied behavior analysis.

[12]  Peng Li,et al.  Biologically inspired reinforcement learning for mobile robot collision avoidance , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[13]  J. Wickens,et al.  A cellular mechanism of reward-related learning , 2001, Nature.

[14]  Karl J. Friston,et al.  Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning , 2004, Science.

[15]  Steve B. Furber,et al.  Large-Scale Simulations of Plastic Neural Networks on Neuromorphic Hardware , 2016, Front. Neuroanat..

[16]  Wulfram Gerstner,et al.  Reinforcement Learning Using a Continuous Time Actor-Critic Framework with Spiking Neurons , 2013, PLoS Comput. Biol..

[17]  Anders Lansner,et al.  Synaptic and nonsynaptic plasticity approximating probabilistic inference , 2014, Front. Synaptic Neurosci..

[18]  Johannes Schemmel,et al.  Reward-based learning under hardware constraints—using a RISC processor embedded in a neuromorphic substrate , 2013, Front. Neurosci..

[19]  M. Colonnier,et al.  Number and size of neurons and synapses in the motor cortex of cats raised in different environmental complexities , 1989, The Journal of comparative neurology.

[20]  P. D. Giudice,et al.  Modelling the formation of working memory with networks of integrate-and-fire neurons connected by plastic synapses , 2003, Journal of Physiology-Paris.

[21]  H. Markram,et al.  Regulation of Synaptic Efficacy by Coincidence of Postsynaptic APs and EPSPs , 1997, Science.

[22]  Markus Diesmann,et al.  Spike-Timing-Dependent Plasticity in Balanced Random Networks , 2007, Neural Computation.

[23]  John E. Stone,et al.  GPU clusters for high-performance computing , 2009, 2009 IEEE International Conference on Cluster Computing and Workshops.

[24]  W. Gerstner,et al.  Triplets of Spikes in a Model of Spike Timing-Dependent Plasticity , 2006, The Journal of Neuroscience.

[25]  L. Abbott,et al.  Competitive Hebbian learning through spike-timing-dependent synaptic plasticity , 2000, Nature Neuroscience.

[26]  E. Izhikevich.  Solving the distal reward problem through linkage of STDP and dopamine signaling , 2007, BMC Neuroscience.

[27]  Steve B. Furber,et al.  Synapse-Centric Mapping of Cortical Models to the SpiNNaker Neuromorphic Architecture , 2016, Front. Neurosci..

[28]  B. Pakkenberg,et al.  Aging and the human neocortex , 2003, Experimental Gerontology.

[29]  Walter Senn,et al.  Learning Real-World Stimuli in a Neural Network with Spike-Driven Synaptic Dynamics , 2007, Neural Computation.

[30]  G. Buzsáki,et al.  The log-dynamic brain: how skewed distributions affect network operations , 2014, Nature Reviews Neuroscience.

[31]  W. Gerstner,et al.  Connectivity reflects coding: a model of voltage-based STDP with homeostasis , 2010, Nature Neuroscience.

[32]  W. Gerstner,et al.  Neuromodulated Spike-Timing-Dependent Plasticity, and Theory of Three-Factor Learning Rules , 2016, Front. Neural Circuits.

[33]  K. Fuxe,et al.  Volume transmission in the CNS and its relevance for neuropsychopharmacology. , 1999, Trends in pharmacological sciences.

[34]  Steve B. Furber,et al.  Power analysis of large-scale, real-time neural networks on SpiNNaker , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[35]  Thomas Nowotny,et al.  GeNN: a code generation framework for accelerated brain simulations , 2016, Scientific Reports.

[36]  Luca Maria Gambardella,et al.  Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition , 2010, ArXiv.

[37]  Andrew P Davison,et al.  Learning Cross-Modal Spatial Transformations through Spike Timing-Dependent Plasticity , 2006, The Journal of Neuroscience.

[38]  Jim D. Garside,et al.  SpiNNaker: A 1-W 18-Core System-on-Chip for Massively-Parallel Neural Network Simulation , 2013, IEEE Journal of Solid-State Circuits.

[39]  Steve B. Furber,et al.  A framework for plasticity implementation on the SpiNNaker neural architecture , 2015, Front. Neurosci..

[40]  Luca Maria Gambardella,et al.  Deep, Big, Simple Neural Nets for Handwritten Digit Recognition , 2010, Neural Computation.

[41]  Daniel J. Amit,et al.  Modeling brain function: the world of attractor neural networks, 1st Edition , 1989 .

[42]  Pierre Yger,et al.  PyNN: A Common Interface for Neuronal Network Simulators , 2008, Front. Neuroinform..

[43]  Prof. Dr. Dr. Valentino Braitenberg,et al.  Cortex: Statistics and Geometry of Neuronal Connectivity , 1998, Springer Berlin Heidelberg.

[44]  C. Gerfen.  Synaptic organization of the striatum. , 1988, Journal of electron microscopy technique.

[45]  P. Garris,et al.  Efflux of dopamine from the synaptic cleft in the nucleus accumbens of the rat brain , 1994, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[46]  Markus Diesmann,et al.  Frontiers in Computational Neuroscience Enabling Functional Neural Circuit Simulations with Distributed Computing of Neuromodulated Plasticity , 2022 .

[47]  Brian Gardner,et al.  Supervised Learning on the SpiNNaker Neuromorphic Hardware , 2017 .

[48]  W. Schultz.  Multiple reward signals in the brain , 2000, Nature Reviews Neuroscience.

[49]  Steve B. Furber,et al.  Accuracy and Efficiency in Fixed-Point Neural ODE Solvers , 2015, Neural Computation.

[50]  Markus Diesmann,et al.  A Spiking Neural Network Model of an Actor-Critic Learning Agent , 2009, Neural Computation.

[51]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[52]  P. Dayan,et al.  Tonic dopamine: opportunity costs and the control of response vigor , 2007, Psychopharmacology.

[53]  P. Greengard,et al.  Dichotomous Dopaminergic Control of Striatal Synaptic Plasticity , 2008, Science.

[54]  Wolfgang Maass,et al.  Emergence of Dynamic Memory Traces in Cortical Microcircuit Models through STDP , 2013, The Journal of Neuroscience.

[55]  Weisong Shi,et al.  Edge Computing: Vision and Challenges , 2016, IEEE Internet of Things Journal.

[56]  G. Bi,et al.  Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type , 1998, The Journal of Neuroscience.

[57]  Markus Diesmann,et al.  An Imperfect Dopaminergic Error Signal Can Drive Temporal-Difference Learning , 2011, PLoS Comput. Biol..

[58]  J. Kerr,et al.  Dopamine Receptor Activation Is Required for Corticostriatal Spike-Timing-Dependent Plasticity , 2008, The Journal of Neuroscience.

[59]  Steve B. Furber,et al.  Implementing spike-timing-dependent plasticity on SpiNNaker neuromorphic hardware , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[60]  Arie E. Kaufman,et al.  GPU Cluster for High Performance Computing , 2004, Proceedings of the ACM/IEEE SC2004 Conference.

[61]  Wulfram Gerstner,et al.  Phenomenological models of synaptic plasticity based on spike timing , 2008, Biological Cybernetics.

[62]  Steve B. Furber,et al.  Breaking the millisecond barrier on SpiNNaker: implementing asynchronous event-based plastic models with microsecond resolution , 2015, Front. Neurosci..