The impact of encoding-decoding schemes and weight normalization in spiking neural networks

Spike-timing Dependent Plasticity (STDP) is a learning mechanism that can capture causal relationships between events. STDP is considered a foundational element of memory and learning in biological neural networks. Previous research efforts endeavored to understand the functionality of STDP's learning window in spiking neural networks (SNNs). In this study, we investigate the interaction among different encoding/decoding schemes, STDP learning windows and normalization rules for the SNN classifier, trained and tested on MNIST, NIST and ETH80-Contour datasets. The results show that when no normalization rules are applied, classical STDP typically achieves the best performance. Additionally, first-spike decoding classifiers require much less decoding time than a spike count decoding classifier. Thirdly, when no normalization rule is applied, the classifier accuracy decreases as the encoding duration increases from 10ms to 34ms using count decoding scheme. Finally, normalization of output weights is shown to improve the performance of a first-spike decoding classifier, which reveals the importance of weight normalization to SNN.

[1]  L. Abbott,et al.  Competitive Hebbian learning through spike-timing-dependent synaptic plasticity , 2000, Nature Neuroscience.

[2]  Yves Frégnac,et al.  A Re-Examination of Hebbian-Covariance Rules and Spike Timing-Dependent Plasticity in Cat Visual Cortex in vivo , 2010, Front. Syn. Neurosci..

[3]  Yoshua Bengio,et al.  Practical Recommendations for Gradient-Based Training of Deep Architectures , 2012, Neural Networks: Tricks of the Trade.

[4]  Timothée Masquelier,et al.  Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity , 2007, PLoS Comput. Biol..

[5]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[6]  J. Mellor,et al.  Frontiers in Synaptic Neuroscience Synaptic Neuroscience Stdp in the Hippocampus: the Data the Activity Requirements for Spike Timing-dependent Plasticity in the Hippocampus , 2022 .

[7]  Filip Ponulak,et al.  Introduction to spiking neural networks: Information processing, learning and applications. , 2011, Acta neurobiologiae experimentalis.

[8]  Dharmendra S. Modha,et al.  Backpropagation for Energy-Efficient Neuromorphic Computing , 2015, NIPS.

[9]  Marti A. Hearst.  Trends & Controversies: Support Vector Machines , 1998, IEEE Intell. Syst..

[10]  José Ranilla,et al.  Particle swarm optimization for hyper-parameter selection in deep neural networks , 2017, GECCO.

[11]  Wulfram Gerstner,et al.  SPIKING NEURON MODELS Single Neurons , Populations , Plasticity , 2002 .

[12]  Simei Gomes Wysoski,et al.  Evolving spiking neural networks for audiovisual information processing , 2010, Neural Networks.

[13]  D J Willshaw,et al.  A marker induction mechanism for the establishment of ordered neural mappings: its application to the retinotectal problem. , 1979, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[14]  G. Bi,et al.  Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type , 1998, The Journal of Neuroscience.

[15]  Nikola Kasabov,et al.  Dynamic evolving spiking neural networks for on-line spatio- and spectro-temporal pattern recognition. , 2013, Neural networks : the official journal of the International Neural Network Society.

[16]  Bruno A. Olshausen,et al.  Book Review , 2003, Journal of Cognitive Neuroscience.

[17]  H. Abarbanel,et al.  Dynamical model of long-term synaptic plasticity , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[18]  Wulfram Gerstner,et al.  Mathematical formulations of Hebbian learning , 2002, Biological Cybernetics.

[19]  Carl E. Rasmussen,et al.  Presynaptic and postsynaptic competition in models for the development of neuromuscular connections , 1993, Biological Cybernetics.

[20]  G. Bi,et al.  Synaptic modification by correlated activity: Hebb's postulate revisited. , 2001, Annual review of neuroscience.

[21]  A. Hendrickson,et al.  Human photoreceptor topography , 1990, The Journal of comparative neurology.

[22]  Y. Dan,et al.  Spike timing-dependent plasticity: a Hebbian learning rule. , 2008, Annual review of neuroscience.

[23]  Gayle M. Wittenberg,et al.  Spike Timing Dependent Plasticity: A Consequence of More Fundamental Learning Rules , 2010, Front. Comput. Neurosci..

[24]  Andrzej J. Kasinski,et al.  Supervised Learning in Spiking Neural Networks with ReSuMe: Sequence Learning, Classification, and Spike Shifting , 2010, Neural Computation.

[25]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[26]  Silvia Scarpetta,et al.  Storage of Phase-Coded Patterns via STDP in Fully-Connected and Sparse Network: A Study of the Network Capacity , 2010, Front. Syn. Neurosci.

[27]  Peter Dayan,et al.  Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems , 2001 .

[28]  Wofgang Maas,et al.  Networks of spiking neurons: the third generation of neural network models , 1997 .

[29]  Romain Brette,et al.  Equation-oriented specification of neural models for simulations , 2013, Front. Neuroinform..

[30]  Wulfram Gerstner,et al.  A History of Spike-Timing-Dependent Plasticity , 2011, Front. Syn. Neurosci..

[31]  P. Jonas,et al.  Symmetric spike timing-dependent plasticity at CA3–CA3 synapses optimizes storage and recall in autoassociative networks , 2016, Nature Communications.

[32]  Eugene M. Izhikevich,et al.  Which model to use for cortical spiking neurons? , 2004, IEEE Transactions on Neural Networks.

[33]  A. Watson.  A formula for human retinal ganglion cell receptive field density as a function of visual field location. , 2014, Journal of vision.

[34]  Eugene M. Izhikevich,et al.  Simple model of spiking neurons , 2003, IEEE Trans. Neural Networks.

[35]  Matthew Cook,et al.  Unsupervised learning of digit recognition using spike-timing-dependent plasticity , 2015, Front. Comput. Neurosci..