A Soft-Pruning Method Applied During Training of Spiking Neural Networks for In-memory Computing Applications

Inspired from the computational efficiency of the biological brain, spiking neural networks (SNNs) emulate biological neural networks, neural codes, dynamics, and circuitry. SNNs show great potential for the implementation of unsupervised learning using in-memory computing. Here, we report an algorithmic optimization that improves energy efficiency of online learning with SNNs on emerging non-volatile memory (eNVM) devices. We develop a pruning method for SNNs by exploiting the output firing characteristics of neurons. Our pruning method can be applied during network training, which is different from previous approaches in the literature that employ pruning on already-trained networks. This approach prevents unnecessary updates of network parameters during training. This algorithmic optimization can complement the energy efficiency of eNVM technology, which offers a unique in-memory computing platform for the parallelization of neural network operations. Our SNN maintains ~90% classification accuracy on the MNIST dataset with up to ~75% pruning, significantly reducing the number of weight updates. The SNN and pruning scheme developed in this work can pave the way toward applications of eNVM based neuro-inspired systems for energy efficient online learning in low power applications.

[1]  Matthew Cook,et al.  Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[2]  Duygu Kuzum,et al.  Drift-Enhanced Unsupervised Learning of Handwritten Digits in Spiking Neural Network With PCM Synapses , 2018, IEEE Electron Device Letters.

[3]  Yasuo Takahashi,et al.  Switching operation and degradation of resistive random access memory composed of tungsten oxide and copper investigated using in-situ TEM , 2015, Scientific Reports.

[4]  Anthony S. Maida,et al.  Multi-layer unsupervised learning in a spiking convolutional neural network , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[5]  Boonserm Kijsirikul,et al.  Decision tree pruning using backpropagation neural networks , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[6]  智晴 長尾,et al.  Deep Neural Network を用いた株式売買戦略の構築 , 2016 .

[7]  Kaushik Roy,et al.  RESPARC: A reconfigurable and energy-efficient architecture with Memristive Crossbars for deep Spiking Neural Networks , 2017, 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC).

[8]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[9]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[10]  Sangheon Oh,et al.  Neuroinspired unsupervised learning and pruning with subquantum CBRAM arrays , 2018, Nature Communications.

[11]  L. Abbott,et al.  Competitive Hebbian learning through spike-timing-dependent synaptic plasticity , 2000, Nature Neuroscience.

[12]  Wulfram Gerstner,et al.  Multicontact Co-operativity in Spike-Timing–Dependent Structural Plasticity Stabilizes Networks , 2016, Cerebral cortex.

[13]  Lyle N. Long,et al.  Character Recognition using Spiking Neural Networks , 2007, 2007 International Joint Conference on Neural Networks.

[14]  Narayan Srinivasa,et al.  Energy-Efficient Neuron, Synapse and STDP Integrated Circuits , 2012, IEEE Transactions on Biomedical Circuits and Systems.

[15]  Myungsoo Kim,et al.  Atomristor: Nonvolatile Resistance Switching in Atomic Sheets of Transition Metal Dichalcogenides. , 2018, Nano letters.

[16]  Kaushik Roy,et al.  Probabilistic Deep Spiking Neural Systems Enabled by Magnetic Tunnel Junction , 2016, IEEE Transactions on Electron Devices.

[17]  Walter Senn,et al.  Learning Real-World Stimuli in a Neural Network with Spike-Driven Synaptic Dynamics , 2007, Neural Computation.

[18]  Wolfgang Maass,et al.  Bayesian Computation Emerges in Generic Cortical Microcircuits through Spike-Timing-Dependent Plasticity , 2013, PLoS Comput. Biol..

[19]  Xuefei Ning,et al.  Fault-tolerant training with on-line fault detection for RRAM-based neural computing systems , 2017, 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC).

[20]  David Kappel,et al.  Synaptic Sampling: A Bayesian Approach to Neural Network Plasticity and Rewiring , 2015, NIPS.

[21]  Kaushik Roy,et al.  Convolutional Spike Timing Dependent Plasticity based Feature Learning in Spiking Neural Networks , 2017, ArXiv.

[22]  Wolfgang Maass,et al.  Networks of Spiking Neurons: The Third Generation of Neural Network Models , 1996, Electron. Colloquium Comput. Complex..

[23]  Chung Lam,et al.  Training a Probabilistic Graphical Model With Resistive Switching Electronic Synapses , 2016, IEEE Transactions on Electron Devices.

[24]  Wofgang Maas,et al.  Networks of spiking neurons: the third generation of neural network models , 1997 .

[25]  Timothée Masquelier,et al.  Acquisition of visual features through probabilistic spike-timing-dependent plasticity , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[26]  Kaushik Roy,et al.  STDP-Based Pruning of Connections and Weight Quantization in Spiking Neural Networks for Energy-Efficient Recognition , 2017, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[27]  H.-S. Philip Wong.  The End of the Road for 2D Scaling of Silicon CMOS and the Future of Device Technology , 2018, 2018 76th Device Research Conference (DRC).

[28]  Matthew Cook,et al.  Unsupervised learning of digit recognition using spike-timing-dependent plasticity , 2015, Front. Comput. Neurosci..

[29]  Fabien Alibart,et al.  Pattern classification by memristive crossbar circuits using ex situ and in situ training , 2013, Nature Communications.

[30]  Farnood Merrikh-Bayat,et al.  Training and operation of an integrated neuromorphic network based on metal-oxide memristors , 2014, Nature.

[31]  Wei Yang Lu,et al.  Nanoscale memristor device as synapse in neuromorphic systems. , 2010, Nano letters.

[32]  Bipin Rajendran,et al.  Spiking neural networks for handwritten digit recognition - Supervised learning and network optimization , 2018, Neural Networks.

[33]  Tobi Delbrück,et al.  Training Deep Spiking Neural Networks Using Backpropagation , 2016, Front. Neurosci..

[34]  Deepak Khosla,et al.  Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition , 2014, International Journal of Computer Vision.

[35]  G. W. Burr,et al.  Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element , 2015, 2014 IEEE International Electron Devices Meeting.

[36]  J. Tenenbaum,et al.  A tutorial introduction to Bayesian models of cognitive development , 2011, Cognition.

[37]  Shimeng Yu,et al.  Low-Energy Robust Neuromorphic Computation Using Synaptic Devices , 2012, IEEE Transactions on Electron Devices.

[38]  Franck Mamalet,et al.  Unsupervised Feature Learning With Winner-Takes-All Based STDP , 2018, Front. Comput. Neurosci..

[39]  Shimeng Yu,et al.  Metal–Oxide RRAM , 2012, Proceedings of the IEEE.

[40]  Gert Cauwenberghs,et al.  Event-driven contrastive divergence for spiking neuromorphic systems , 2013, Front. Neurosci..

[41]  Jiaming Zhang,et al.  Analogue signal and image processing with large memristor crossbars , 2017, Nature Electronics.

[42]  Xiaochen Peng,et al.  NeuroSim: A Circuit-Level Macro Model for Benchmarking Neuro-Inspired Architectures in Online Learning , 2018, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[43]  Wei D. Lu,et al.  Data Clustering using Memristor Networks , 2015, Scientific Reports.

[44]  Byoungil Lee,et al.  Nanoelectronic programmable synapses based on phase change materials for brain-inspired computing. , 2012, Nano letters.

[45]  Gert Cauwenberghs,et al.  Memristors Empower Spiking Neurons With Stochasticity , 2015, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[46]  Jinseok Kim,et al.  Deep Neural Network Optimized to Resistive Memory with Nonlinear Current-Voltage Characteristics , 2017, ACM J. Emerg. Technol. Comput. Syst..

[47]  Kaushik Roy,et al.  Magnetic Tunnel Junction Based Long-Term Short-Term Stochastic Synapse for a Spiking Neural Network with On-Chip STDP Learning , 2016, Scientific Reports.

[48]  Richard George,et al.  Structural Plasticity Denoises Responses and Improves Learning Speed , 2016, Front. Comput. Neurosci..

[49]  Giacomo Indiveri,et al.  Beyond spike-timing dependent plasticity in memristor crossbar arrays , 2016, 2016 IEEE International Symposium on Circuits and Systems (ISCAS).

[50]  Alessandro E. P. Villa,et al.  Effect of stimulus-driven pruning on the detection of spatiotemporal patterns of activity in large neural networks , 2007, Biosyst..

[51]  S. Thorpe,et al.  STDP-based spiking deep convolutional neural networks for object recognition , 2018 .

[52]  Ali Khiat,et al.  Unsupervised learning in probabilistic neural networks with multi-state metal-oxide memristive synapses , 2016, Nature Communications.

[53]  Moritz Helias,et al.  Spike-Timing Dependence of Structural Plasticity Explains Cooperative Synapse Formation in the Neocortex , 2012, PLoS Comput. Biol..

[54]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[55]  Jacques-Olivier Klein,et al.  Spin-transfer torque magnetic memory as a stochastic memristive synapse , 2014, 2014 IEEE International Symposium on Circuits and Systems (ISCAS).

[56]  Kaushik Roy,et al.  EnsembleSNN: Distributed assistive STDP learning for energy-efficient recognition in spiking neural networks , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).