MorphIC: A 65-nm 738k-Synapse/mm$^2$ Quad-Core Binary-Weight Digital Neuromorphic Processor With Stochastic Spike-Driven Online Learning

Recent trends in the field of neural network accelerators investigate weight quantization as a means to increase the resource- and power-efficiency of hardware devices. As full on-chip weight storage is necessary to avoid the high energy cost of off-chip memory accesses, memory reduction requirements for weight storage pushed toward the use of binary weights, which were demonstrated to have a limited accuracy reduction on many applications when quantization-aware training techniques are used. In parallel, spiking neural network (SNN) architectures are explored to further reduce power when processing sparse event-based data streams, while on-chip spike-based online learning appears as a key feature for applications constrained in power and resources during the training phase. However, designing power- and area-efficient SNNs still requires the development of specific techniques in order to leverage on-chip online learning on binary weights without compromising the synapse density. In this paper, we demonstrate MorphIC, a quad-core binary-weight digital neuromorphic processor embedding a stochastic version of the spike-driven synaptic plasticity (S-SDSP) learning rule and a hierarchical routing fabric for large-scale chip interconnection. The MorphIC SNN processor embeds a total of 2k leaky integrate-and-fire (LIF) neurons and more than two million plastic synapses for an active silicon area of 2.86 mm$^2$ in 65-nm CMOS, achieving a high density of 738k synapses/mm$^2$. MorphIC demonstrates an order-of-magnitude improvement in the area-accuracy tradeoff on the MNIST classification task compared to previously-proposed SNNs, while having no penalty in the energy-accuracy tradeoff.

[1]  Danielle Smith Bassett,et al.  Small-World Brain Networks , 2006, The Neuroscientist : a review journal bringing neurobiology, neurology and psychiatry.

[2]  Steve B. Furber,et al.  Understanding the interconnection network of SpiNNaker , 2009, ICS.

[3]  David Bol,et al.  A 65-nm 738k-Synapse/mm2 Quad-Core Binary-Weight Digital Neuromorphic Processor with Stochastic Spike-Driven Online Learning , 2019, 2019 IEEE International Symposium on Circuits and Systems (ISCAS).

[4]  David Bol,et al.  A 0.086-mm$^2$ 12.7-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28-nm CMOS , 2018, IEEE Transactions on Biomedical Circuits and Systems.

[5]  Marian Verhelst,et al.  Minimum energy quantized neural networks , 2017, 2017 51st Asilomar Conference on Signals, Systems, and Computers.

[6]  Giacomo Indiveri,et al.  A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses , 2015, Front. Neurosci..

[7]  Mostafa Rahimi Azghadi,et al.  Spike-Based Synaptic Plasticity in Silicon: Design, Implementation, Application, and Challenges , 2014, Proceedings of the IEEE.

[8]  René Schüffny,et al.  A Biological-Realtime Neuromorphic System in 28 nm CMOS Using Low-Leakage Switched Capacitor Circuits , 2014, IEEE Transactions on Biomedical Circuits and Systems.

[9]  Marian Verhelst,et al.  An always-on 3.8μJ/86% CIFAR-10 mixed-signal binary CNN processor with all memory on chip in 28nm CMOS , 2018, 2018 IEEE International Solid - State Circuits Conference - (ISSCC).

[10]  Boris Murmann,et al.  BinarEye: An always-on energy-accuracy-scalable binary CNN processor with all memory on chip in 28nm CMOS , 2018, 2018 IEEE Custom Integrated Circuits Conference (CICC).

[11]  Eric A. Vittoz,et al.  A communication architecture tailored for analog VLSI artificial neural networks: intrinsic performance and limitations , 1994, IEEE Trans. Neural Networks.

[12]  Gregory K. Chen,et al.  A 4096-Neuron 1M-Synapse 3.8PJ/SOP Spiking Neural Network with On-Chip STDP Learning and Sparse Weights in 10NM FinFET CMOS , 2018, 2018 IEEE Symposium on VLSI Circuits.

[13]  Zhengya Zhang,et al.  A Sparse Coding Neural Network ASIC With On-Chip Learning for Feature Extraction and Encoding , 2015, IEEE Journal of Solid-State Circuits.

[14]  Keshab K. Parhi,et al.  VLSI digital signal processing systems , 1999 .

[15]  Hesham Mostafa,et al.  Supervised Learning Based on Temporal Coding in Spiking Neural Networks , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[16]  Arindam Basu,et al.  A 2.86-TOPS/W Current Mirror Cross-Bar-Based Machine-Learning and Physical Unclonable Function Engine For Internet-of-Things Applications , 2019, IEEE Transactions on Circuits and Systems I: Regular Papers.

[17]  Walter Senn,et al.  Learning Real-World Stimuli in a Neural Network with Spike-Driven Synaptic Dynamics , 2007, Neural Computation.

[18]  Ran El-Yaniv,et al.  Binarized Neural Networks , 2016, NIPS.

[19]  Ran El-Yaniv,et al.  Binarized Neural Networks , 2016, ArXiv.

[20]  Gu-Yeon Wei,et al.  14.3 A 28nm SoC with a 1.2GHz 568nJ/prediction sparse deep-neural-network engine with >0.1 timing error rate tolerance for IoT applications , 2017, 2017 IEEE International Solid-State Circuits Conference (ISSCC).

[21]  Keshab K. Parhi,et al.  High-Speed Parallel CRC Implementation Based on Unfolding, Pipelining, and Retiming , 2006, IEEE Transactions on Circuits and Systems II: Express Briefs.

[22]  Luca Benini,et al.  YodaNN: An Architecture for Ultralow Power Binary-Weight CNN Acceleration , 2016, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[23]  David Bol,et al.  A 0.086-mm2 12.7-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28-nm CMOS , 2019, IEEE Trans. Biomed. Circuits Syst..

[24]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[25]  Yong Liu,et al.  A 45nm CMOS neuromorphic chip with a scalable architecture for learning in networks of spiking neurons , 2011, 2011 IEEE Custom Integrated Circuits Conference (CICC).

[26]  Surya Ganguli,et al.  SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks , 2017, Neural Computation.

[27]  Bernard Brezzo,et al.  TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip , 2015, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[28]  Yann LeCun,et al.  The mnist database of handwritten digits , 2005 .

[29]  Barbara De Salvo Brain-Inspired technologies: Towards chips that think? , 2018, 2018 IEEE International Solid - State Circuits Conference - (ISSCC).

[30]  Rodrigo Alvarez-Icaza,et al.  Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural Simulations , 2014, Proceedings of the IEEE.

[31]  Zhengya Zhang,et al.  A 640M pixel/s 3.65mW sparse event-driven neuromorphic object recognition processor with on-chip learning , 2015, 2015 Symposium on VLSI Circuits (VLSI Circuits).

[32]  Somnath Paul,et al.  Event-Driven Random Back-Propagation: Enabling Neuromorphic Deep Learning Machines , 2016, Front. Neurosci..

[33]  G. Indiveri,et al.  Neuromorphic architectures for spiking deep neural networks , 2015, 2015 IEEE International Electron Devices Meeting (IEDM).

[34]  David Bol,et al.  A fully-synthesized 20-gate digital spike-based synapse with embedded online learning , 2017, 2017 IEEE International Symposium on Circuits and Systems (ISCAS).

[35]  Michael P. Flynn,et al.  A 3.43TOPS/W 48.9pJ/pixel 50.1nJ/classification 512 analog neuron sparse coding neural network with on-chip learning and classification in 40nm CMOS , 2017, 2017 Symposium on VLSI Circuits.

[36]  Timothy P Lillicrap,et al.  Towards deep learning with segregated dendrites , 2016, eLife.

[37]  Dharmendra S. Modha,et al.  Backpropagation for Energy-Efficient Neuromorphic Computing , 2015, NIPS.

[38]  W. Senn,et al.  Convergence of stochastic learning in perceptrons with binary synapses. , 2005, Physical review. E, Statistical, nonlinear, and soft matter physics.

[39]  Kwabena Boahen,et al.  Point-to-point connectivity between neuromorphic chips using address events , 2000 .

[40]  Bernabé Linares-Barranco,et al.  On Practical Issues for Stochastic STDP Hardware With 1-bit Synaptic Weights , 2018, Front. Neurosci..

[41]  Giacomo Indiveri,et al.  A Scalable Multicore Architecture With Heterogeneous Memory Structures for Dynamic Neuromorphic Asynchronous Processors (DYNAPs) , 2017, IEEE Transactions on Biomedical Circuits and Systems.

[42]  Hong Wang,et al.  Loihi: A Neuromorphic Manycore Processor with On-Chip Learning , 2018, IEEE Micro.

[43]  Gert Cauwenberghs,et al.  Hierarchical Address Event Routing for Reconfigurable Large-Scale Neuromorphic Systems , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[44]  Johannes Schemmel,et al.  A wafer-scale neuromorphic hardware system for large-scale neural modeling , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[45]  Chi-Sang Poon,et al.  Neuromorphic Silicon Neurons and Large-Scale Neural Networks: Challenges and Opportunities , 2011, Front. Neurosci..

[46]  Pinaki Mazumder,et al.  Online Supervised Learning for Hardware-Based Multilayer Spiking Neural Networks Through the Modulation of Weight-Dependent Spike-Timing-Dependent Plasticity , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[47]  Giacomo Indiveri,et al.  A neuromorphic systems approach to in-memory computing with non-ideal memristive devices: From mitigation to exploitation , 2018, Faraday discussions.

[48]  Arindam Basu,et al.  A Learning-Enabled Neuron Array IC Based Upon Transistor Channel Models of Biological Phenomena , 2013, IEEE Transactions on Biomedical Circuits and Systems.

[49]  Jim D. Garside,et al.  SpiNNaker: A 1-W 18-Core System-on-Chip for Massively-Parallel Neural Network Simulation , 2013, IEEE Journal of Solid-State Circuits.