Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations

Spiking Neural Networks (SNNs) play an important role in neuroscience as they help neuroscientists understand how the nervous system works. To model the nervous system, SNNs incorporate the concept of time into neurons and inter-neuron interactions called spikes; a neuron's internal state changes with respect to time and input spikes, and a neuron fires an output spike when its internal state satisfies certain conditions. As the neurons forming the nervous system behave differently, SNN simulation frameworks must be able to simulate the diverse behaviors of the neurons. To support any neuron models, some frameworks rely on general purpose processors at the cost of inefficiency in simulation speed and energy consumption. The other frameworks employ specialized accelerators to overcome the inefficiency; however, the accelerators support only a limited set of neuron models due to their model-driven designs, making accelerator-based frameworks unable to simulate target SNNs. In this paper, we present Flexon, a flexible digital neuron which exploits the biologically common features shared by diverse neuron models, to enable efficient SNN simulations. To design Flexon, we first collect SNNs from prior work in neuroscience research and analyze the neuron models the SNNs employ. From the analysis, we observe that the neuron models share a set of biologically common features, and that the features can be combined to simulate a significantly larger set of neuron behaviors than the existing model-driven designs. Furthermore, we find that the features share a small set of computational primitives which can be exploited to further reduce the chip area. The resulting digital neurons, Flexon and spatially folded Flexon, are flexible, highly efficient, and can be easily integrated with existing hardware. Our prototyping results using TSMC 45 nm standard cell library show that a 12-neuron Flexon array improves energy efficiency by 6,186x and 422x over CPU and GPU, respectively, in a small footprint of 9.26 mm2. The results also show that a 72-neuron spatially folded Flexon array incurs a smaller footprint of 7.62 mm2 and achieves geomean speedups of 122.45x and 9.83x over CPU and GPU, respectively.

[1]  Murray Shanahan,et al.  NeMo: A Platform for Neural Modelling of Spiking Neurons Using GPUs , 2009, 2009 20th IEEE International Conference on Application-specific Systems, Architectures and Processors.

[2]  A. Hodgkin,et al.  A quantitative description of membrane current and its application to conduction and excitation in nerve , 1990 .

[3]  James Hetherington,et al.  Computational challenges of systems biology , 2004, Computer.

[4]  Ramón Huerta,et al.  Self-organization in the olfactory system: one shot odor recognition in insects , 2005, Biological Cybernetics.

[5]  M L Hines,et al.  Neuron: A Tool for Neuroscientists , 2001, The Neuroscientist : a review journal bringing neurobiology, neurology and psychiatry.

[6]  Jim D. Garside,et al.  Overview of the SpiNNaker System Architecture , 2013, IEEE Transactions on Computers.

[7]  Norman P. Jouppi,et al.  CACTI: an enhanced cache access and cycle time model , 1996, IEEE J. Solid State Circuits.

[8]  Alain Destexhe,et al.  Self-sustained Asynchronous Irregular States and Up–down States in Thalamic, Cortical and Thalamocortical Networks of Nonlinear Integrate-and-fire Neurons , 2022 .

[9]  Jim D. Garside,et al.  SpiNNaker: A 1-W 18-Core System-on-Chip for Massively-Parallel Neural Network Simulation , 2013, IEEE Journal of Solid-State Circuits.

[10]  Rajeev Balasubramonian,et al.  INXS: Bridging the throughput and energy gap for spiking neural networks , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[11]  E. Fehlberg,et al.  Low-order classical Runge-Kutta formulas with stepsize control and their application to some heat transfer problems , 1969 .

[12]  James E. Smith,et al.  Efficient digital neurons for large scale cortical architectures , 2014, 2014 ACM/IEEE 41st International Symposium on Computer Architecture (ISCA).

[13]  Yong Liu,et al.  A 45nm CMOS neuromorphic chip with a scalable architecture for learning in networks of spiking neurons , 2011, 2011 IEEE Custom Integrated Circuits Conference (CICC).

[14]  Wayne Luk,et al.  NeuroFlow: A General Purpose Spiking Neural Network Simulation Platform using Customizable Processors , 2016, Front. Neurosci..

[15]  Nikil D. Dutt,et al.  Efficient Spiking Neural Network Model of Pattern Motion Selectivity in Visual Cortex , 2014, Neuroinformatics.

[16]  Deepak Khosla,et al.  Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition , 2014, International Journal of Computer Vision.

[17]  Peter Dayan,et al.  Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems , 2001 .

[18]  Wofgang Maas,et al.  Networks of spiking neurons: the third generation of neural network models , 1997 .

[19]  R. Stein.  A THEORETICAL ANALYSIS OF NEURONAL VARIABILITY. , 1965, Biophysical journal.

[20]  Matthew Cook,et al.  Unsupervised learning of digit recognition using spike-timing-dependent plasticity , 2015, Front. Comput. Neurosci..

[21]  Nicol N. Schraudolph,et al.  A Fast, Compact Approximation of the Exponential Function , 1999, Neural Computation.

[22]  Nikil D. Dutt,et al.  A configurable simulation environment for the efficient simulation of large-scale spiking neural networks on graphics processors , 2009, Neural Networks.

[23]  Andrew S. Cassidy,et al.  Building block of a programmable neuromorphic substrate: A digital neurosynaptic core , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[24]  Tobias C. Potjans,et al.  The Cell-Type Specific Cortical Microcircuit: Relating Structure and Activity in a Full-Scale Spiking Network Model , 2012, Cerebral cortex.

[25]  Johannes Schemmel,et al.  Methods for Simulating High-Conductance States in Neural Microcircuits , 2004 .

[26]  Jim D. Garside,et al.  SpiNNaker: Design and Implementation of a GALS Multicore System-on-Chip , 2011, JETC.

[27]  Mikko H. Lipasti,et al.  Bridging the semantic gap: Emulating biological neuronal behaviors with simple digital neurons , 2013, 2013 IEEE 19th International Symposium on High Performance Computer Architecture (HPCA).

[28]  Robert A. Legenstein,et al.  Neuromorphic hardware in the loop: Training a deep spiking network on the BrainScaleS wafer-scale system , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[29]  Bernard Brezzo,et al.  TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip , 2015, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[30]  Lei Zhang,et al.  Neuromorphic accelerators: A comparison between neuroscience and machine-learning approaches , 2015, 2015 48th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

[31]  Tim P Vogels,et al.  Signal Propagation and Logic Gating in Networks of Integrate-and-Fire Neurons , 2005, The Journal of Neuroscience.

[32]  Eugene M. Izhikevich,et al.  Simple model of spiking neurons , 2003, IEEE Trans. Neural Networks.

[33]  Andres Upegui,et al.  An FPGA platform for on-line topology exploration of spiking neural networks , 2005, Microprocess. Microsystems.

[34]  Marc-Oliver Gewaltig,et al.  NEST (NEural Simulation Tool) , 2007, Scholarpedia.

[35]  Rodrigo Alvarez-Icaza,et al.  Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural Simulations , 2014, Proceedings of the IEEE.

[36]  D. Hansel,et al.  How Spike Generation Mechanisms Determine the Neuronal Response to Fluctuating Inputs , 2003, The Journal of Neuroscience.

[37]  Damien Querlioz,et al.  Extraction of temporally correlated features from dynamic vision sensors with spike-timing-dependent plasticity , 2012, Neural Networks.

[38]  Yannick Bornat,et al.  Biorealistic spiking neural network on FPGA , 2013, 2013 47th Annual Conference on Information Sciences and Systems (CISS).

[39]  Romain Brette,et al.  The Brian Simulator , 2009, Front. Neurosci..

[40]  Nicholas T. Carnevale,et al.  Simulation of networks of spiking neurons: A review of tools and strategies , 2006, Journal of Computational Neuroscience.

[41]  Romain Brette,et al.  Equation-oriented specification of neural models for simulations , 2013, Front. Neuroinform..

[42]  Nikil Dutt,et al.  An efficient automated parameter tuning framework for spiking neural networks , 2014, Front. Neurosci..

[43]  Henning Sprekeler,et al.  Inhibitory Plasticity Balances Excitation and Inhibition in Sensory Pathways and Memory Networks , 2011, Science.

[44]  Hojjat Adeli,et al.  Spiking Neural Networks , 2009, Int. J. Neural Syst..

[45]  Eugene M. Izhikevich,et al.  Which model to use for cortical spiking neurons? , 2004, IEEE Transactions on Neural Networks.

[46]  Nicolas Brunel,et al.  Dynamics of Sparsely Connected Networks of Excitatory and Inhibitory Spiking Neurons , 2000, Journal of Computational Neuroscience.

[47]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[48]  Luis A. Plana,et al.  SpiNNaker: Mapping neural networks onto a massively-parallel chip multiprocessor , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[49]  Nikil D. Dutt,et al.  An Efficient Simulation Environment for Modeling Large-Scale Cortical Processing , 2011, Front. Neuroinform..

[50]  Timothée Masquelier,et al.  Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity , 2007, PLoS Comput. Biol..

[51]  Andrew S. Cassidy,et al.  Cognitive computing building block: A versatile and efficient digital neuron model for neurosynaptic cores , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[52]  James E. Smith,et al.  Space-Time Computing with Temporal Neural Networks , 2017, Synthesis Lectures on Computer Architecture.

[53]  Thomas Nowotny,et al.  GeNN: a code generation framework for accelerated brain simulations , 2016, Scientific Reports.

[54]  Nikil D. Dutt,et al.  CARLsim 3: A user-friendly and highly optimized library for the creation of neurobiologically detailed spiking neural networks , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[55]  Abhishek Bhattacharjee,et al.  Using Branch Predictors to Predict Brain Activity in Brain-Machine Implants , 2017, 2017 50th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

[56]  Steve B. Furber,et al.  The SpiNNaker Project , 2014, Proceedings of the IEEE.

[57]  Johannes Schemmel,et al.  A wafer-scale neuromorphic hardware system for large-scale neural modeling , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[58]  Wulfram Gerstner,et al.  Limits to high-speed simulations of spiking neural networks using general-purpose computers , 2014, Front. Neuroinform..

[59]  Wulfram Gerstner,et al.  Adaptive exponential integrate-and-fire model as an effective description of neuronal activity. , 2005, Journal of neurophysiology.

[60]  Narayan Srinivasa,et al.  Energy-Efficient Neuron, Synapse and STDP Integrated Circuits , 2012, IEEE Transactions on Biomedical Circuits and Systems.

[61]  Pierre Yger,et al.  PyNN: A Common Interface for Neuronal Network Simulators , 2008, Front. Neuroinform..

[62]  Wulfram Gerstner,et al.  Neuronal Dynamics: From Single Neurons To Networks And Models Of Cognition , 2014 .