A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses

Implementing compact, low-power artificial neural processing systems with real-time on-line learning abilities is still an open challenge. In this paper we present a full-custom mixed-signal VLSI device with neuromorphic learning circuits that emulate the biophysics of real spiking neurons and dynamic synapses for exploring the properties of computational neuroscience models and for building brain-inspired computing systems. The proposed architecture allows the on-chip configuration of a wide range of network connectivities, including recurrent and deep networks, with short-term and long-term plasticity. The device comprises 128 K analog synapse and 256 neuron circuits with biologically plausible dynamics and bi-stable spike-based plasticity mechanisms that endow it with on-line learning abilities. In addition to the analog circuits, the device comprises also asynchronous digital logic circuits for setting different synapse and neuron properties as well as different network configurations. This prototype device, fabricated using a 180 nm 1P6M CMOS process, occupies an area of 51.4 mm2, and consumes approximately 4 mW for typical experiments, for example involving attractor networks. Here we describe the details of the overall architecture and of the individual circuits and present experimental results that showcase its potential. By supporting a wide range of cortical-like computational modules comprising plasticity mechanisms, this device will enable the realization of intelligent autonomous systems with on-line learning capabilities.

[1]  Chiara Bartolozzi,et al.  Ultra low leakage synaptic scaling circuits for implementing homeostatic plasticity in neuromorphic architectures , 2014, 2014 IEEE International Symposium on Circuits and Systems (ISCAS).

[2]  Shih-Chii Liu,et al.  Modeling Short-Term Synaptic Depression in Silicon , 2003, Neural Computation.

[3]  Timothée Masquelier,et al.  Competitive STDP-Based Spike Pattern Learning , 2009, Neural Computation.

[4]  Walter Senn,et al.  Learning Only When Necessary: Better Memories of Correlated Patterns in Networks with Bounded Synapses , 2005, Neural Computation.

[5]  Kwabena Boahen,et al.  Optic nerve signals in a neuromorphic chip II: testing and results , 2004, IEEE Transactions on Biomedical Engineering.

[6]  Giacomo Indiveri,et al.  A VLSI array of low-power spiking neurons and bistable synapses with spike-timing dependent plasticity , 2006, IEEE Transactions on Neural Networks.

[7]  X. Wang,et al.  Synaptic Basis of Cortical Persistent Activity: the Importance of NMDA Receptors to Working Memory , 1999, The Journal of Neuroscience.

[8]  Bernabé Linares-Barranco,et al.  On Real-Time AER 2-D Convolutions Hardware for Neuromorphic Spike-Based Cortical Processing , 2008, IEEE Transactions on Neural Networks.

[9]  Rodrigo Alvarez-Icaza,et al.  Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural Simulations , 2014, Proceedings of the IEEE.

[10]  Rajit Manohar.  Reconfigurable Asynchronous Logic , 2006, IEEE Custom Integrated Circuits Conference 2006.

[11]  Giacomo Indiveri,et al.  PyNCS: a microkernel for high-level definition and configuration of neuromorphic electronic systems , 2014, Front. Neuroinform..

[12]  Ueli Rutishauser,et al.  State-Dependent Computation Using Coupled Recurrent Networks , 2008, Neural Computation.

[13]  Chiara Bartolozzi,et al.  Neuromorphic Electronic Circuits for Building Autonomous Cognitive Systems , 2014, Proceedings of the IEEE.

[14]  Michael Schmuker,et al.  A neuromorphic network for generic multivariate data classification , 2014, Proceedings of the National Academy of Sciences.

[15]  Daniel J. Amit,et al.  Spike-Driven Synaptic Dynamics Generating Working Memory States , 2003, Neural Computation.

[16]  Chiara Bartolozzi,et al.  Synaptic Dynamics in Analog VLSI , 2007, Neural Computation.

[17]  Giacomo Indiveri,et al.  Exploiting device mismatch in neuromorphic VLSI systems to implement axonal delays , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[18]  C. Mead,et al.  Neuromorphic analogue VLSI. , 1995, Annual review of neuroscience.

[19]  Tobi Delbruck,et al.  Real-time classification and sensor fusion with a spiking deep belief network , 2013, Front. Neurosci..

[20]  Paul E. Hasler,et al.  A bio-physically inspired silicon neuron , 2004, IEEE Transactions on Circuits and Systems I: Regular Papers.

[21]  Chiara Bartolozzi,et al.  A selective attention multi--chip system with dynamic synapses and spiking neurons , 2006, NIPS.

[22]  Wulfram Gerstner,et al.  Adaptive exponential integrate-and-fire model , 2009, Scholarpedia.

[23]  Shih-Chii Liu,et al.  Neuromorphic sensory systems , 2010, Current Opinion in Neurobiology.

[24]  L. Abbott,et al.  Synaptic plasticity: taming the beast , 2000, Nature Neuroscience.

[25]  Gregory Cohen,et al.  An FPGA Implementation of a Polychronous Spiking Neural Network with Delay Adaptation , 2013, Front. Neurosci..

[26]  Jennifer Hasler,et al.  Neuron Array With Plastic Synapses and Programmable Dendrites , 2013, IEEE Transactions on Biomedical Circuits and Systems.

[27]  Kea-Tiong Tang,et al.  VLSI Implementation of a Bio-Inspired Olfactory Spiking Neural Network , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[28]  Kwabena Boahen,et al.  Point-to-point connectivity between neuromorphic chips using address events , 2000 .

[29]  Giacomo Indiveri,et al.  Emergent Auditory Feature Tuning in a Real-Time Neuromorphic VLSI System , 2011, Front. Neurosci..

[30]  Shih-Chii Liu,et al.  Analog VLSI: Circuits and Principles , 2002 .

[31]  Rodney J. Douglas,et al.  A pulse-coded communications infrastructure for neuromorphic systems , 1999 .

[32]  Camille Couprie,et al.  Learning Hierarchical Features for Scene Labeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Wolfgang Maass,et al.  STDP enables spiking neurons to detect hidden causes of their inputs , 2009, NIPS.

[34]  Shih-Chii Liu.  Analog VLSI Circuits for Short-Term Dynamic Synapses , 2003, EURASIP J. Adv. Signal Process..

[35]  Mark C. W. van Rossum,et al.  Memory retention and spike-timing-dependent plasticity. , 2009, Journal of neurophysiology.

[36]  Marc'Aurelio Ranzato,et al.  Building high-level features using large scale unsupervised learning , 2011, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[37]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[38]  Walter Senn,et al.  Learning Real-World Stimuli in a Neural Network with Spike-Driven Synaptic Dynamics , 2007, Neural Computation.

[39]  Steve B. Furber,et al.  The SpiNNaker Project , 2014, Proceedings of the IEEE.

[40]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[41]  Rahul Sarpeshkar,et al.  An analog VLSI cochlea with new transconductance amplifiers and nonlinear gain control , 1996, 1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96.

[42]  Bernabé Linares-Barranco,et al.  On the design and characterization of femtoampere current-mode circuits , 2003, IEEE J. Solid State Circuits.

[43]  Alex M. Andrew,et al.  Boosting: Foundations and Algorithms , 2012 .

[44]  Analog Vlsi,et al.  On the Design of , 2000 .

[45]  Patrick Camilleri,et al.  A VLSI network of spiking neurons with plastic fully configurable “stop-learning” synapses , 2008, 2008 15th IEEE International Conference on Electronics, Circuits and Systems.

[46]  P. D. Giudice,et al.  Modelling the formation of working memory with networks of integrate-and-fire neurons connected by plastic synapses , 2003, Journal of Physiology-Paris.

[47]  Walter Senn,et al.  Beyond spike timing: the role of nonlinear plasticity and unreliable synapses , 2002, Biological Cybernetics.

[48]  Giacomo Indiveri,et al.  Spatio-temporal Spike Pattern Classification in Neuromorphic Systems , 2013, Living Machines.

[49]  R Meddis,et al.  Analog very large-scale integrated (VLSI) implementation of a model of amplitude-modulation sensitivity in the auditory brainstem. , 1999, The Journal of the Acoustical Society of America.

[50]  Patrick Camilleri,et al.  Robust Working Memory in an Asynchronously Spiking Neural Network Realized with Neuromorphic VLSI , 2011, Frontiers in Neuroscience.

[51]  Bernabé Linares-Barranco,et al.  A Spatial Contrast Retina With On-Chip Calibration for Neuromorphic Spike-Based AER Vision Systems , 2007, IEEE Transactions on Circuits and Systems I: Regular Papers.

[52]  Peng Xu,et al.  Stochastic Synapse with Short-Term Depression for Silicon Neurons , 2007, 2007 IEEE Biomedical Circuits and Systems Conference.

[53]  S. Joshi,et al.  65k-neuron integrate-and-fire array transceiver with address-event reconfigurable synaptic routing , 2012, 2012 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[54]  Tobi Delbrück,et al.  32-bit Configurable bias current generator with sub-off-current capability , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[55]  Tobi Delbruck,et al.  Robotic goalie with 3 ms reaction time at 4% CPU load using event-based dynamic vision sensor , 2013, Front. Neurosci..

[56]  Johannes Schemmel,et al.  Characterization and Compensation of Network-Level Anomalies in Mixed-Signal Neuromorphic Modeling Platforms , 2014, PloS one.

[57]  David Burr,et al.  Suppression of the magnocellular pathway during saccades , 1996, Behavioural Brain Research.

[58]  Richard H. R. Hahnloser,et al.  Silicon synaptic depression , 2001, Biological Cybernetics.

[59]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[60]  Kwabena Boahen,et al.  Optic nerve signals in a neuromorphic chip I: Outer and inner retina models , 2004, IEEE Transactions on Biomedical Engineering.

[61]  W. Gerstner,et al.  Spike-Timing-Dependent Plasticity: A Comprehensive Overview , 2012, Front. Syn. Neurosci..

[62]  Stefano Fusi,et al.  The Sparseness of Mixed Selectivity Neurons Controls the Generalization–Discrimination Trade-Off , 2013, The Journal of Neuroscience.

[63]  N. Spruston,et al.  Questions about STDP as a General Model of Synaptic Plasticity , 2010, Front. Syn. Neurosci..

[64]  Giacomo Indiveri,et al.  Synthesizing cognition in neuromorphic electronic systems , 2013, Proceedings of the National Academy of Sciences.

[65]  Craig T. Jin,et al.  A log-domain implementation of the Mihalas-Niebur neuron model , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[66]  R. Douglas,et al.  A silicon neuron , 1991, Nature.

[67]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[68]  Mattia Rigotti,et al.  A Simple Derivation of a Bound on the Perceptron Margin Using Singular Value Decomposition , 2011, Neural Computation.

[69]  Geoffrey E. Hinton,et al.  Acoustic Modeling Using Deep Belief Networks , 2012, IEEE Transactions on Audio, Speech, and Language Processing.

[70]  Daniel J. Amit,et al.  Modeling brain function: the world of attractor neural networks, 1st Edition , 1989 .

[71]  Shih-Chii Liu,et al.  Minitaur, an Event-Driven FPGA-Based Spiking Network Accelerator , 2014, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[72]  Eugene M. Izhikevich,et al.  Simple model of spiking neurons , 2003, IEEE Trans. Neural Networks.

[73]  Kwabena Boahen,et al.  Thermodynamically Equivalent Silicon Models of Voltage-Dependent Ion Channels , 2007, Neural Computation.

[74]  Stefano Fusi,et al.  Attractor concretion as a mechanism for the formation of context representations , 2010, NeuroImage.

[75]  LeCunYann,et al.  Learning Hierarchical Features for Scene Labeling , 2013 .

[76]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[77]  Craig T. Jin,et al.  A log-domain implementation of the Izhikevich neuron model , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[78]  Johannes Schemmel,et al.  A comprehensive workflow for general-purpose neural modeling with highly configurable neuromorphic hardware systems , 2010, Biological Cybernetics.

[79]  Gert Cauwenberghs,et al.  Neuromorphic Silicon Neuron Circuits , 2011, Front. Neurosci.

[80]  Wulfram Gerstner,et al.  Adaptive exponential integrate-and-fire model as an effective description of neuronal activity. , 2005, Journal of neurophysiology.