Visual explanations from spiking neural networks using inter-spike intervals

By emulating biological features in brain, Spiking Neural Networks (SNNs) offer an energy-efficient alternative to conventional deep learning. To make SNNs ubiquitous, a ‘visual explanation’ technique for analysing and explaining the internal spike behavior of such temporal deep SNNs is crucial. Explaining SNNs visually will make the network more transparent giving the end-user a tool to understand how SNNs make temporal predictions and why they make a certain decision. In this paper, we propose a bio-plausible visual explanation tool for SNNs, called Spike Activation Map (SAM). SAM yields a heatmap (i.e., localization map) corresponding to each time-step of input data by highlighting neurons with short inter-spike interval activity. Interestingly, without the use of gradients and ground truth, SAM produces a temporal localization map highlighting the region of interest in an image attributed to an SNN’s prediction at each time-step. Overall, SAM outsets the beginning of a new research area ‘explainable neuromorphic computing’ that will ultimately allow end-users to establish appropriate trust in predictions from SNNs.

[1]  Sungroh Yoon,et al.  Fast and Efficient Information Transmission with Burst Spikes in Deep Spiking Neural Networks , 2018, 2019 56th ACM/IEEE Design Automation Conference (DAC).

[2]  Steve B. Furber,et al.  The SpiNNaker Project , 2014, Proceedings of the IEEE.

[3]  T. Masquelier,et al.  S4NN: temporal backpropagation for spiking neural networks with one spike per neuron , 2019, Int. J. Neural Syst..

[4]  Kaushik Roy,et al.  Towards spike-based machine intelligence with neuromorphic computing , 2019, Nature.

[5]  Forrest N. Iandola,et al.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size , 2016, ArXiv.

[6]  Kaushik Roy,et al.  Going Deeper in Spiking Neural Networks: VGG and Residual Architectures , 2018, Front. Neurosci..

[7]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  E. Adrian,et al.  The impulses produced by sensory nerve-endings: Part II. The response of a Single End-Organ. , 2006, The Journal of physiology.

[9]  A. B. Bonds,et al.  Burst firing and modulation of functional connectivity in cat striate cortex. , 1998, Journal of neurophysiology.

[10]  Matthew Cook,et al.  Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[11]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[12]  Leslie G. Ungerleider,et al.  Mechanisms of visual attention in the human cortex. , 2000, Annual review of neuroscience.

[13]  Priyadarshini Panda,et al.  Revisiting Batch Normalization for Training Low-latency Deep Spiking Neural Networks from Scratch , 2020, ArXiv.

[14]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[15]  Emre Neftci,et al.  Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks , 2019, IEEE Signal Processing Magazine.

[16]  Jan van Gemert,et al.  Zoom-CAM: Generating Fine-grained Pixel Annotations from Image Labels , 2020, 2020 25th International Conference on Pattern Recognition (ICPR).

[17]  Gouhei Tanaka,et al.  2021 Roadmap on Neuromorphic Computing and Engineering , 2021, ArXiv.

[18]  Minsuk Kahng,et al.  Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers , 2018, IEEE Transactions on Visualization and Computer Graphics.

[19]  G. Bi,et al.  Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type , 1998, The Journal of Neuroscience.

[20]  Malu Zhang,et al.  Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks. , 2020 .

[21]  Kaushik Roy,et al.  Towards Scalable, Efficient and Accurate Deep Spiking Neural Networks with Backward Residual Connections, Stochastic Softmax and Hybridization , 2019, ArXiv.

[22]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[23]  Jeffrey L. Krichmar,et al.  A self-driving robot using deep convolutional neural networks on neuromorphic hardware , 2016, 2017 International Joint Conference on Neural Networks (IJCNN).

[24]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[25]  Max Welling,et al.  Visualizing Deep Neural Network Decisions: Prediction Difference Analysis , 2017, ICLR.

[26]  Deepak Khosla,et al.  Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition , 2014, International Journal of Computer Vision.

[27]  J. Knott.  The organization of behavior: A neuropsychological theory , 1951 .

[28]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[29]  Kiyoung Choi,et al.  Deep neural networks with weighted spikes , 2018, Neurocomputing.

[30]  Leslie G. Ungerleider,et al.  Mechanisms of directed attention in the human extrastriate cortex as revealed by functional MRI. , 1998, Science.

[31]  Tobi Delbrück,et al.  Training Deep Spiking Neural Networks Using Backpropagation , 2016, Front. Neurosci..

[32]  Shih-Chii Liu,et al.  Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification , 2017, Front. Neurosci..

[33]  J. Lisman.  Bursts as a unit of neural information: making unreliable synapses reliable , 1997, Trends in Neurosciences.

[34]  Bernard Brezzo,et al.  TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip , 2015, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[35]  Thomas Brox,et al.  Inverting Visual Representations with Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Zijian Zhang,et al.  Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[37]  Hesham Mostafa,et al.  Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks , 2019, IEEE Signal Processing Magazine.

[38]  Kaushik Roy,et al.  RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[40]  Leslie G. Ungerleider,et al.  The neural basis of biased competition in human visual cortex , 2001, Neuropsychologia.

[41]  Hong Wang,et al.  Loihi: A Neuromorphic Manycore Processor with On-Chip Learning , 2018, IEEE Micro.

[42]  Eugene M. Izhikevich,et al.  Simple model of spiking neurons , 2003, IEEE Trans. Neural Networks.

[43]  Akshayvarun Subramanya,et al.  Fooling Network Interpretation in Image Classification , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[44]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[45]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Kaushik Roy,et al.  Inherent Adversarial Robustness of Deep Spiking Neural Networks: Effects of Discrete Input Encoding and Non-Linear Activations , 2020, ECCV.

[47]  F. Mechler,et al.  Interspike Intervals, Receptive Fields, and Information Encoding in Primary Visual Cortex , 2000, The Journal of Neuroscience.

[48]  C. Atencio,et al.  Improved stimulus representation by short interspike intervals in primary auditory cortex. , 2011, Journal of neurophysiology.

[49]  Matthew Cook,et al.  Unsupervised learning of digit recognition using spike-timing-dependent plasticity , 2015, Front. Comput. Neurosci..

[50]  Yuan Xie,et al.  Rethinking the performance comparison between SNNS and ANNS , 2020, Neural Networks.

[51]  Eli Shlizerman,et al.  Neural Interactome: Interactive Simulation of a Neuronal System , 2017, bioRxiv.

[52]  Lei Deng,et al.  Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks , 2017, Front. Neurosci..

[53]  Vivienne Sze,et al.  Efficient Processing of Deep Neural Networks: A Tutorial and Survey , 2017, Proceedings of the IEEE.

[54]  Jyrki Alakuijala,et al.  Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function , 2019, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[55]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[56]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[58]  R. Reid,et al.  Precisely correlated firing in cells of the lateral geniculate nucleus , 1996, Nature.

[59]  Colin J. Akerman,et al.  Random synaptic feedback weights support error backpropagation for deep learning , 2016, Nature Communications.

[60]  Thomas L. Dean,et al.  Neural Networks and Neuroscience-Inspired Computer Vision , 2014, Current Biology.

[61]  Changick Kim,et al.  Combinational Class Activation Maps for Weakly Supervised Object Localization , 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[62]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  Dmitry Nekhaev,et al.  Recurrent Spiking Neural Network Learning Based on a Competitive Maximization of Neuronal Activity , 2018, Front. Neuroinform..

[64]  Priyadarshini Panda,et al.  A Comprehensive Analysis on Adversarial Robustness of Spiking Neural Networks , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[65]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[66]  Kaushik Roy,et al.  Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures , 2019, Frontiers in Neuroscience.

[67]  Yann LeCun,et al.  The mnist database of handwritten digits , 2005 .

[68]  Alois Knoll,et al.  A Survey of Robotics Control Based on Learning-Inspired Spiking Neural Networks , 2018, Front. Neurorobot..

[69]  Stephen Grossberg,et al.  Competitive Learning: From Interactive Activation to Adaptive Resonance , 1987, Cogn. Sci..

[70]  Hesham Mostafa,et al.  Supervised Learning Based on Temporal Coding in Spiking Neural Networks , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[71]  Antonio Torralba,et al.  HOGgles: Visualizing Object Detection Features , 2013, 2013 IEEE International Conference on Computer Vision.

[72]  Giacomo Indiveri,et al.  Parameter Optimization and Learning in a Spiking Neural Network for UAV Obstacle Avoidance Targeting Neuromorphic Processors , 2019, IEEE Transactions on Neural Networks and Learning Systems.