Structural plasticity on an accelerated analog neuromorphic hardware system

In computational neuroscience, as well as in machine learning, neuromorphic devices promise an accelerated and scalable alternative to neural network simulations. Their neural connectivity and synaptic capacity depend on their specific design choices, but is always intrinsically limited. Here, we present a strategy to achieve structural plasticity that optimizes resource allocation under these constraints by constantly rewiring the pre- and postsynaptic partners while keeping the neuronal fan-in constant and the connectome sparse. In particular, we implemented this algorithm on the analog neuromorphic system BrainScaleS-2. It was executed on a custom embedded digital processor located on chip, accompanying the mixed-signal substrate of spiking neurons and synapse circuits. We evaluated our implementation in a simple supervised learning scenario, showing its ability to optimize the network topology with respect to the nature of its training data, as well as its overall computational efficiency.

[1]  Richard George,et al.  Activity dependent structural plasticity in neuromorphic systems , 2017, 2017 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[2]  Steve B. Furber,et al.  Structural Plasticity on the SpiNNaker Many-Core Neuromorphic System , 2018, Front. Neurosci..

[3]  Johannes Schemmel,et al.  A wafer-scale neuromorphic hardware system for large-scale neural modeling , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[4]  Joseph E LeDoux,et al.  Structural plasticity and memory , 2004, Nature Reviews Neuroscience.

[5]  Johannes Schemmel,et al.  An analog dynamic memory array for neuromorphic hardware , 2013, 2013 European Conference on Circuit Theory and Design (ECCTD).

[6]  W. Gan,et al.  Dendritic spine dynamics. , 2009, Annual review of physiology.

[7]  Steve B. Furber,et al.  Efficient Reward-Based Structural Plasticity on a SpiNNaker 2 Prototype , 2019, IEEE Transactions on Biomedical Circuits and Systems.

[8]  Richard George,et al.  Structural Plasticity Denoises Responses and Improves Learning Speed , 2016, Front. Comput. Neurosci..

[9]  G. Shepherd,et al.  Transient and Persistent Dendritic Spines in the Neocortex In Vivo , 2005, Neuron.

[10]  Johannes Schemmel,et al.  A Mixed-Signal Structured AdEx Neuron for Accelerated Neuromorphic Cores , 2018, IEEE Transactions on Biomedical Circuits and Systems.

[11]  Gert Cauwenberghs,et al.  Large-Scale Neuromorphic Spiking Array Processors: A Quest to Mimic the Brain , 2018, Front. Neurosci..

[12]  N. Kasthuri,et al.  Long-term dendritic spine stability in the adult cortex , 2002, Nature.

[13]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[14]  Johannes Schemmel,et al.  An Accelerated LIF Neuronal Network Array for a Large-Scale Mixed-Signal Neuromorphic Architecture , 2018, IEEE Transactions on Circuits and Systems I: Regular Papers.

[15]  W. Gan,et al.  Development of Long-Term Dendritic Spine Stability in Diverse Regions of Cerebral Cortex , 2005, Neuron.

[16]  Surya Ganguli,et al.  SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks , 2017, Neural Computation.

[17]  Bernard Brezzo,et al.  TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip , 2015, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[18]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[19]  David Bol,et al.  A 0.086-mm$^2$ 12.7-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28-nm CMOS , 2018, IEEE Transactions on Biomedical Circuits and Systems.

[20]  Bartlett W. Mel,et al.  Impact of Active Dendrites and Structural Plasticity on the Memory Capacity of Neural Tissue , 2001, Neuron.

[21]  David Bol,et al.  A 0.086-mm2 12.7-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28-nm CMOS , 2019, IEEE Trans. Biomed. Circuits Syst..

[22]  Hesham Mostafa,et al.  Supervised Learning Based on Temporal Coding in Spiking Neural Networks , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[23]  F. Sommer,et al.  Structural Plasticity, Effectual Connectivity, and Memory in Cortex , 2016, Front. Neuroanat..

[24]  Yiran Chen,et al.  Learning Structured Sparsity in Deep Neural Networks , 2016, NIPS.

[25]  Yasushi Miyashita,et al.  Dendritic spine geometry is critical for AMPA receptor expression in hippocampal CA1 pyramidal neurons , 2001, Nature Neuroscience.

[26]  Johannes Schemmel,et al.  Demonstrating Hybrid Learning in a Flexible Neuromorphic Hardware System , 2016, IEEE Transactions on Biomedical Circuits and Systems.

[27]  Johannes Schemmel,et al.  Modeling Synaptic Plasticity within Networks of Highly Accelerated I&F Neurons , 2007, 2007 IEEE International Symposium on Circuits and Systems.

[28]  David Kappel,et al.  Synaptic Sampling: A Bayesian Approach to Neural Network Plasticity and Rewiring , 2015, NIPS.

[29]  Johannes Schemmel,et al.  Demonstrating Advantages of Neuromorphic Computation: A Pilot Study , 2018, Front. Neurosci..

[30]  Giacomo Indiveri,et al.  A Scalable Multicore Architecture With Heterogeneous Memory Structures for Dynamic Neuromorphic Asynchronous Processors (DYNAPs) , 2017, IEEE Transactions on Biomedical Circuits and Systems.

[31]  W. Gerstner,et al.  Neuromodulated Spike-Timing-Dependent Plasticity, and Theory of Three-Factor Learning Rules , 2016, Front. Neural Circuits.

[32]  F. Wörgötter,et al.  Activity-dependent structural plasticity , 2009, Brain Research Reviews.

[33]  Shaista Hussain,et al.  Multiclass Classification by Adaptive Network of Dendritic Neurons with Binary Synapses Using Structural Plasticity , 2016, Front. Neurosci..

[34]  Misha Denil,et al.  Predicting Parameters in Deep Learning , 2014 .

[35]  Steve B. Furber,et al.  Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype , 2018, Front. Neurosci..

[36]  Gert Cauwenberghs,et al.  Event-driven contrastive divergence for spiking neuromorphic systems , 2013, Front. Neurosci..

[37]  K. Svoboda,et al.  Experience-dependent structural synaptic plasticity in the mammalian brain , 2009, Nature Reviews Neuroscience.

[38]  Subhrajit Roy,et al.  An Online Unsupervised Structural Plasticity Algorithm for Spiking Neural Networks , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[39]  Willie F. Tobin,et al.  Rapid formation and selective stabilization of synapses for enduring motor memories , 2009, Nature.

[40]  Michael Schmuker,et al.  A neuromorphic network for generic multivariate data classification , 2014, Proceedings of the National Academy of Sciences.

[41]  Karel Svoboda,et al.  Experience-dependent and cell-type-specific spine growth in the neocortex , 2006, Nature.

[42]  Viola Priesemann,et al.  Control of criticality and computation in spiking neuromorphic networks with plasticity , 2020, Nature Communications.

[43]  Karlheinz Meier,et al.  Neuromorphic Hardware Learns to Learn , 2019, Front. Neurosci..

[44]  Hong Wang,et al.  Loihi: A Neuromorphic Manycore Processor with On-Chip Learning , 2018, IEEE Micro.

[45]  David Kappel,et al.  Deep Rewiring: Training very sparse deep networks , 2017, ICLR.

[46]  K. Svoboda,et al.  Long-term in vivo imaging of experience-dependent synaptic plasticity in adult cortex , 2002, Nature.

[47]  W. Senn,et al.  Learning by the Dendritic Prediction of Somatic Spiking , 2014, Neuron.

[48]  Subhrajit Roy,et al.  Spiking Neural Classifier with Lumped Dendritic Nonlinearity and Binary Synapses: A Current Mode VLSI Implementation and Analysis , 2018, Neural Computation.

[49]  Jim D. Garside,et al.  Overview of the SpiNNaker System Architecture , 2013, IEEE Transactions on Computers.

[50]  Subhrajit Roy,et al.  Liquid State Machine With Dendritically Enhanced Readout for Low-Power, Neuromorphic VLSI Implementations , 2014, IEEE Transactions on Biomedical Circuits and Systems.

[51]  Robert A. Legenstein,et al.  Neuromorphic hardware in the loop: Training a deep spiking network on the BrainScaleS wafer-scale system , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).