Constructing Accurate and Efficient Deep Spiking Neural Networks with Double-threshold and Augmented Schemes

Spiking neural networks (SNNs) are considered as a potential candidate to overcome current challenges, such as the high-power consumption encountered by artificial neural networks (ANNs); however, there is still a gap between them with respect to the recognition accuracy on various tasks. A conversion strategy was, thus, introduced recently to bridge this gap by mapping a trained ANN to an SNN. However, it is still unclear that to what extent this obtained SNN can benefit both the accuracy advantage from ANN and high efficiency from the spike-based paradigm of computation. In this article, we propose two new conversion methods, namely TerMapping and AugMapping. The TerMapping is a straightforward extension of a typical threshold-balancing method with a double-threshold scheme, while the AugMapping additionally incorporates a new scheme of augmented spike that employs a spike coefficient to carry the number of typical all-or-nothing spikes occurring at a time step. We examine the performance of our methods based on the MNIST, Fashion-MNIST, and CIFAR10 data sets. The results show that the proposed double-threshold scheme can effectively improve the accuracies of the converted SNNs. More importantly, the proposed AugMapping is more advantageous for constructing accurate, fast, and efficient deep SNNs compared with other state-of-the-art approaches. Our study, therefore, provides new approaches for further integration of advanced techniques in ANNs to improve the performance of SNNs, which could be of great merit to applied developments with spike-based neuromorphic computing.

[1]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[2]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[3]  Wolfgang Maass,et al.  Networks of Spiking Neurons: The Third Generation of Neural Network Models , 1996, Electron. Colloquium Comput. Complex..

[4]  Kaushik Roy,et al.  Towards spike-based machine intelligence with neuromorphic computing , 2019, Nature.

[5]  Timothée Masquelier,et al.  Deep Learning in Spiking Neural Networks , 2018, Neural Networks.

[6]  Bin Deng,et al.  Training Spiking Neural Networks for Cognitive Tasks: A Versatile Framework Compatible to Various Temporal Codes , 2017, ArXiv.

[7]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[8]  Lei Deng,et al.  Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks , 2017, Front. Neurosci..

[9]  Gopalakrishnan Srinivasan,et al.  Deep Spiking Convolutional Neural Network Trained With Unsupervised Spike-Timing-Dependent Plasticity , 2019, IEEE Transactions on Cognitive and Developmental Systems.

[10]  Robert A. Nawrocki,et al.  A Mini Review of Neuromorphic Architectures and Implementations , 2016, IEEE Transactions on Electron Devices.

[11]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[12]  Matthew Cook,et al.  Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[13]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[14]  Ammar Belatreche,et al.  A Supervised Learning Algorithm for Learning Precise Timing of Multiple Spikes in Multilayer Spiking Neural Networks , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[16]  Wulfram Gerstner,et al.  SPIKING NEURON MODELS Single Neurons , Populations , Plasticity , 2002 .

[17]  Anand Raghunathan,et al.  Dynamic Spike Bundling for Energy-Efficient Spiking Neural Networks , 2019, 2019 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED).

[18]  S. Thorpe,et al.  STDP-based spiking deep convolutional neural networks for object recognition , 2018 .

[19]  Bo Xu,et al.  A Biologically Plausible Supervised Learning Method for Spiking Neural Networks Using the Symmetric STDP Rule , 2018, Neural Networks.

[20]  Sander M. Bohte,et al.  Error-backpropagation in temporally encoded networks of spiking neurons , 2000, Neurocomputing.

[21]  Deepak Khosla,et al.  Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition , 2014, International Journal of Computer Vision.

[22]  Andrew L. Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[23]  Haizhou Li,et al.  A Spiking Neural Network System for Robust Sequence Recognition , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[24]  Wenrui Zhang,et al.  Hybrid Macro/Micro Level Backpropagation for Training Deep Spiking Neural Networks , 2018, NeurIPS.

[25]  Garrick Orchard,et al.  SLAYER: Spike Layer Error Reassignment in Time , 2018, NeurIPS.

[26]  Timothée Masquelier,et al.  First-Spike-Based Visual Categorization Using Reward-Modulated STDP , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[27]  Hesham Mostafa,et al.  Supervised Learning Based on Temporal Coding in Spiking Neural Networks , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[28]  Tianqi Chen,et al.  Empirical Evaluation of Rectified Activations in Convolutional Network , 2015, ArXiv.

[29]  Ruizhi Chen,et al.  Fast and Efficient Deep Sparse Multi-Strength Spiking Neural Networks with Dynamic Pruning , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[30]  Bernabé Linares-Barranco,et al.  Mapping from Frame-Driven to Frame-Free Event-Driven Vision Systems by Low-Rate Rate Coding and Coincidence Processing--Application to Feedforward ConvNets , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Sungroh Yoon,et al.  Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection , 2020, AAAI.

[32]  Wofgang Maas,et al.  Networks of spiking neurons: the third generation of neural network models , 1997 .

[33]  Andrés Montoyo,et al.  Advances on natural language processing , 2007, Data Knowl. Eng..

[34]  J. Feldmann,et al.  All-optical spiking neurosynaptic networks with self-learning capabilities , 2019, Nature.

[35]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[36]  Michael Pfeiffer,et al.  Deep Learning With Spiking Neurons: Opportunities and Challenges , 2018, Front. Neurosci..

[37]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[38]  G. Bi,et al.  Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type , 1998, The Journal of Neuroscience.

[39]  H. Sompolinsky,et al.  The tempotron: a neuron that learns spike timing–based decisions , 2006, Nature Neuroscience.

[40]  Song Han,et al.  Trained Ternary Quantization , 2016, ICLR.

[41]  Sungroh Yoon,et al.  Fast and Efficient Information Transmission with Burst Spikes in Deep Spiking Neural Networks , 2018, 2019 56th ACM/IEEE Design Automation Conference (DAC).

[42]  Steve B. Furber,et al.  The SpiNNaker Project , 2014, Proceedings of the IEEE.

[43]  Geoffrey Zweig,et al.  Toward Human Parity in Conversational Speech Recognition , 2017, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[44]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[45]  Yoshua Bengio,et al.  BinaryConnect: Training Deep Neural Networks with binary weights during propagations , 2015, NIPS.

[46]  Craig M. Vineyard,et al.  Training deep neural networks for binary communication with the Whetstone method , 2019 .

[47]  Shih-Chii Liu,et al.  Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification , 2017, Front. Neurosci..

[48]  Hesham Mostafa,et al.  Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks , 2019, IEEE Signal Processing Magazine.

[49]  Ali Farhadi,et al.  XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks , 2016, ECCV.

[50]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[51]  Kiyoung Choi,et al.  Deep neural networks with weighted spikes , 2018, Neurocomputing.

[52]  Huajin Tang,et al.  STCA: Spatio-Temporal Credit Assignment with Delayed Feedback in Deep Spiking Neural Networks , 2019, IJCAI.

[53]  Hong Wang,et al.  Loihi: A Neuromorphic Manycore Processor with On-Chip Learning , 2018, IEEE Micro.

[54]  Haizhou Li,et al.  Spike Timing or Rate? Neurons Learn to Make Decisions for Both Through Threshold-Driven Plasticity , 2019, IEEE Transactions on Cybernetics.

[55]  Ammar Belatreche,et al.  A review of learning in biologically plausible spiking neural networks , 2019, Neural Networks.

[56]  Kaushik Roy,et al.  Going Deeper in Spiking Neural Networks: VGG and Residual Architectures , 2018, Front. Neurosci..

[57]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[58]  Matthew Cook,et al.  Unsupervised learning of digit recognition using spike-timing-dependent plasticity , 2015, Front. Comput. Neurosci..

[59]  Haizhou Li,et al.  Rapid Feedforward Computation by Temporal Encoding and Learning With Spiking Neurons , 2013, IEEE Transactions on Neural Networks and Learning Systems.