Multi-layer unsupervised learning in a spiking convolutional neural network

Spiking neural networks (SNNs) have advantages over traditional, non-spiking networks with respect to biorealism, potential for low-power hardware implementations, and theoretical computing power. However, in practice, spiking networks with multi-layer learning have proven difficult to train. This paper explores a novel, bio-inspired spiking convolutional neural network (CNN) that is trained in a greedy, layer-wise fashion. The spiking CNN consists of a convolutional/pooling layer followed by a feature discovery layer, both of which undergo bio-inspired learning. Kernels for the convolutional layer are trained using a sparse, spiking auto-encoder representing primary visual features. The feature discovery layer uses a probabilistic spike-timing-dependent plasticity (STDP) learning rule. This layer represents complex visual features using WTA-thresholded, leaky, integrate-and-fire (LIF) neurons. The new model is evaluated on the MNIST digit dataset using clean and noisy images. Intermediate results show that the convolutional layer is stack-admissible, enabling it to support a multi-layer learning architecture. The recognition performance for clean images is above 98%. This performance is accounted for by the independent and informative visual features extracted in a hierarchy of convolutional and feature discovery layers. The performance loss for recognizing the noisy images is in the range 0.1% to 8.5%. This level of performance loss indicates that the network is robust to additive noise.

[1]  Michael Robert DeWeese,et al.  A Sparse Coding Model with Synaptically Local Plasticity and Spiking Neurons Can Account for the Diverse Shapes of V1 Simple Cell Receptive Fields , 2011, PLoS Comput. Biol..

[2]  Jean-Pascal Pfister,et al.  Sequence learning with hidden units in spiking neural networks , 2011, NIPS.

[3]  Yann LeCun,et al.  Learning Invariant Feature Hierarchies , 2012, ECCV Workshops.

[4]  Tara N. Sainath,et al.  Deep convolutional neural networks for LVCSR , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[5]  Tobi Delbrück,et al.  Training Deep Spiking Neural Networks Using Backpropagation , 2016, Front. Neurosci..

[6]  Nikil D. Dutt,et al.  Categorization and decision-making in a neurobiologically plausible spiking network using a STDP-like learning rule , 2013, Neural Networks.

[7]  Jochen Triesch,et al.  Independent Component Analysis in Spiking Neurons , 2010, PLoS Comput. Biol..

[8]  Wolfgang Maass,et al.  Neural Dynamics as Sampling: A Model for Stochastic Computation in Recurrent Networks of Spiking Neurons , 2011, PLoS Comput. Biol..

[9]  Anthony S. Maida,et al.  Training a Hidden Markov Model with a Bayesian Spiking Neural Network , 2016, Journal of Signal Processing Systems.

[10]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[11]  Timothée Masquelier,et al.  STDP-based spiking deep neural networks for object recognition , 2016, Neural Networks.

[12]  Peter Dayan,et al.  Probabilistic Computation in Spiking Populations , 2004, NIPS.

[13]  Kendra S. Burbank.  Mirrored STDP Implements Autoencoder Learning in a Network of Spiking Neurons , 2015, PLoS Comput. Biol..

[14]  Wolfgang Maass,et al.  Emergence of Dynamic Memory Traces in Cortical Microcircuit Models through STDP , 2013, The Journal of Neuroscience.

[15]  Wolfgang Maass,et al.  STDP enables spiking neurons to detect hidden causes of their inputs , 2009, NIPS.

[16]  Simei Gomes Wysoski,et al.  Fast and adaptive network of spiking neurons for multi-view visual pattern recognition , 2008, Neurocomputing.

[17]  P. Földiák,et al.  Forming sparse representations by local anti-Hebbian learning , 1990, Biological Cybernetics.

[18]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[19]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[20]  Kaushik Roy,et al.  Unsupervised regenerative learning of hierarchical features in Spiking Deep Networks for object recognition , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[21]  Tobi Delbruck,et al.  Real-time classification and sensor fusion with a spiking deep belief network , 2013, Front. Neurosci..

[22]  Y. Dan,et al.  Spike timing-dependent plasticity: from synapse to perception. , 2006, Physiological reviews.

[23]  Wolfgang Maass,et al.  Learning Probabilistic Inference through Spike-Timing-Dependent Plasticity123 , 2016, eNeuro.

[24]  Joel Zylberberg,et al.  Inhibitory Interneurons Decorrelate Excitatory Cells to Drive Sparse Code Formation in a Spiking Model of V1 , 2013, The Journal of Neuroscience.

[25]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[26]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[27]  Ammar Belatreche,et al.  Bio-inspired hierarchical framework for multi-view face detection and pose estimation , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[28]  Deepak Khosla,et al.  Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition , 2014, International Journal of Computer Vision.

[29]  Timothée Masquelier,et al.  Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity , 2007, PLoS Comput. Biol..

[30]  Gert Cauwenberghs,et al.  Event-driven contrastive divergence for spiking neuromorphic systems , 2013, Front. Neurosci..

[31]  Timothée Masquelier,et al.  Acquisition of visual features through probabilistic spike-timing-dependent plasticity , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[32]  Matthew Cook,et al.  Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[33]  Katsuhiko Mori,et al.  Convolutional spiking neural network model for robust face detection , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[34]  Timothée Masquelier,et al.  Bio-inspired unsupervised learning of visual features leads to robust invariant object recognition , 2015, Neurocomputing.