An Online Unsupervised Structural Plasticity Algorithm for Spiking Neural Networks

In this paper, we propose a novel winner-take-all (WTA) architecture employing neurons with nonlinear dendrites and an online unsupervised structural plasticity rule for training it. Furthermore, to aid hardware implementations, our network employs only binary synapses. The proposed learning rule is inspired by spike-timing-dependent plasticity but differs for each dendrite based on its activation level. It trains the WTA network through formation and elimination of connections between inputs and synapses. To demonstrate the performance of the proposed network and learning rule, we employ it to solve two-class, four-class, and six-class classification of random Poisson spike time inputs. The results indicate that by proper tuning of the inhibitory time constant of the WTA, a tradeoff between specificity and sensitivity of the network can be achieved. We use the inhibitory time constant to set the number of subpatterns per pattern we want to detect. We show that while the percentages of successful trials are 92%, 88%, and 82% for two-class, four-class, and six-class classification when no pattern subdivisions are made, it increases to 100% when each pattern is subdivided into 5 or 10 subpatterns. However, the former scenario of no pattern subdivision is more jitter resilient than the later ones.

[1]  Shih-Chii Liu,et al.  A Normalizing aVLSI Network with Controllable Winner-Take-All Properties , 2002 .

[2]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[3]  M. Yoshioka.  Spike-timing-dependent learning rule to encode spatiotemporal patterns in a network of spiking neurons. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[4]  Timothée Masquelier,et al.  Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity , 2007, PLoS Comput. Biol..

[5]  Giacomo Indiveri,et al.  A Current-Mode Hysteretic Winner-take-all Network, with Excitatory and Inhibitory Coupling , 2001 .

[6]  Arnaud Delorme,et al.  Networks of integrate-and-fire neurons using Rank Order Coding B: Spike timing dependent plasticity and emergence of orientation selectivity , 2001, Neurocomputing.

[7]  Henry Markram,et al.  Computer models and analysis tools for neural microcircuits , 2003 .

[8]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[9]  Arindam Basu,et al.  A Learning-Enabled Neuron Array IC Based Upon Transistor Channel Models of Biological Phenomena , 2013, IEEE Transactions on Biomedical Circuits and Systems.

[10]  Shih-Chii Liu,et al.  Computation with Spikes in a Winner-Take-All Network , 2009, Neural Computation.

[11]  Shaista Hussain,et al.  Morphological learning: Increased memory capacity of neuromorphic systems with binary synapses exploiting AER based reconfiguration , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[12]  Bartlett W. Mel,et al.  An Augmented Two-Layer Model Captures Nonlinear Analog Spatial Integration Effects in Pyramidal Neuron Dendrites , 2014, Proceedings of the IEEE.

[13]  Kwabena Boahen,et al.  Point-to-point connectivity between neuromorphic chips using address events , 2000 .

[14]  Janusz A. Starzyk,et al.  CMOS current mode winner-take-all circuit with both excitatory and inhibitory feedback , 1993 .

[15]  Bernabé Linares-Barranco,et al.  A modular current-mode high-precision winner-take-all circuit , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[16]  Edgar Sanchez-Sinencio,et al.  Min-net winner-take-all CMOS implementation , 1993 .

[17]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[18]  Samuel Kaski,et al.  Winner-take-all networks for physiological models of competitive learning , 1994, Neural Networks.

[19]  Rufin VanRullen,et al.  Temporal codes and sparse representations: A key to understanding rapid processing in the visual system , 2004, Journal of Physiology-Paris.

[20]  Samy Bengio,et al.  The Handbook of Brain Theory and Neural Networks , 2002 .

[21]  David Kappel,et al.  STDP Installs in Winner-Take-All Circuits an Online Approximation to Hidden Markov Model Learning , 2014, PLoS Comput. Biol..

[22]  Wolfgang Maass,et al.  Neural Computation with Winner-Take-All as the Only Nonlinear Operation , 1999, NIPS.

[23]  Timothée Masquelier,et al.  Competitive STDP-Based Spike Pattern Learning , 2009, Neural Computation.

[24]  Shaista Hussain,et al.  Hardware-Amenable Structural Learning for Spike-Based Pattern Classification Using a Simple Model of Active Dendrites , 2014, Neural Computation.

[25]  Subhrajit Roy,et al.  A current-mode spiking neural classifier with lumped dendritic nonlinearity , 2015, 2015 IEEE International Symposium on Circuits and Systems (ISCAS).

[26]  Wulfram Gerstner,et al.  Why spikes? Hebbian learning and retrieval of time-resolved excitation patterns , 1993, Biological Cybernetics.

[27]  Giacomo Indiveri,et al.  Winner-Take-All Networks with Lateral Excitation , 1997 .

[28]  Bartlett W. Mel,et al.  Impact of Active Dendrites and Structural Plasticity on the Memory Capacity of Neural Tissue , 2001, Neuron.

[29]  Subhrajit Roy,et al.  Architectural exploration for on-chip, online learning in spiking neural networks , 2014, 2014 International Symposium on Integrated Circuits (ISIC).

[30]  Shaista Hussain,et al.  Hardware efficient, neuromorphic dendritically enhanced readout for liquid state machines , 2013, 2013 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[31]  Gerald M. Edelman,et al.  Temporal sequence learning in winner-take-all networks of spiking neurons demonstrated in a brain-based device , 2013, Front. Neurorobot..

[32]  John A. Barnden,et al.  Temporal winner-take-all networks: a time-based mechanism for fast selection in neural networks , 1993, IEEE Trans. Neural Networks.

[33]  T. Poggio,et al.  Hierarchical models of object recognition in cortex , 1999, Nature Neuroscience.

[34]  Wolfgang Maass,et al.  Bayesian Computation Emerges in Generic Cortical Microcircuits through Spike-Timing-Dependent Plasticity , 2013, PLoS Comput. Biol..

[35]  Michael A. Arbib,et al.  The handbook of brain theory and neural networks , 1995, A Bradford book.

[36]  Subhrajit Roy,et al.  Liquid State Machine With Dendritically Enhanced Readout for Low-Power, Neuromorphic VLSI Implementations , 2014, IEEE Transactions on Biomedical Circuits and Systems.

[37]  Wolfgang Maass,et al.  On the Computational Power of Winner-Take-All , 2000, Neural Computation.