CAVIAR: A 45k Neuron, 5M Synapse, 12G Connects/s AER Hardware Sensory–Processing– Learning–Actuating System for High-Speed Visual Object Recognition and Tracking

This paper describes CAVIAR, a massively parallel hardware implementation of a spike-based sensing-processing-learning-actuating system inspired by the physiology of the nervous system. CAVIAR uses the asynchronous address-event representation (AER) communication framework and was developed in the context of a European Union funded project. It has four custom mixed-signal AER chips, five custom digital AER interface components, 45 k neurons (spiking cells), up to 5 M synapses, performs 12 G synaptic operations per second, and achieves millisecond object recognition and tracking latencies.

[1]  Jack Sklansky,et al.  Finding circles by an array of accumulators , 1975, Commun. ACM.

[2]  G. Shepherd.  The Synaptic Organization of the Brain , 1979 .

[3]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[4]  T R Crimmins,et al.  Geometric filter for speckle reduction. , 1985, Applied optics.

[5]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[6]  Teuvo Kohonen,et al.  Self-organization and associative memory: 3rd edition , 1989 .

[7]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[8]  Kunihiko Fukushima,et al.  Analysis of the process of visual pattern recognition by the neocognitron , 1989, Neural Networks.

[9]  T. Sacktor.  The Synaptic Organization of the Brain (3rd Ed.) , 1991 .

[10]  Carver A. Mead,et al.  Scanners for visualizing activity of analog VLSI circuitry , 1991 .

[11]  Misha Anne Mahowald,et al.  VLSI analogs of neuronal visual processing: a synthesis of form and function , 1992 .

[12]  John Wawrzynek,et al.  Silicon Auditory Processors as Computer Peripherals , 1992, NIPS.

[13]  Massimo A. Sivilotti,et al.  Wiring considerations in analog VLSI systems, with application to field-programmable networks , 1992 .

[14]  Misha A. Mahowald,et al.  An Analog VLSI System for Stereoscopic Vision , 1994 .

[15]  Eric A. Vittoz,et al.  A communication architecture tailored for analog VLSI artificial neural networks: intrinsic performance and limitations , 1994, IEEE Trans. Neural Networks.

[16]  Stephen Grossberg,et al.  Synthetic aperture radar processing by a multiple scale neural system for boundary and surface representation , 1995, Neural Networks.

[17]  Eric A. Vittoz,et al.  A communication scheme for analog VLSI perceptive systems , 1995 .

[18]  John Wawrzynek,et al.  A multi-sender asynchronous extension to the AER protocol , 1995, Proceedings Sixteenth Conference on Advanced Research in VLSI.

[19]  Kazuyuki Aihara,et al.  Dynamical Cell Assembly Hypothesis -- Theoretical Possibility of Spatio-temporal Coding in the Cortex , 1996, Neural Networks.

[20]  Denis Fize,et al.  Speed of processing in the human visual system , 1996, Nature.

[21]  S. Thorpe,et al.  Speed of processing in the human visual system , 1996, Nature.

[22]  Andreas G. Andreou,et al.  Asynchronous Communication of 2D Motion Information Using Winner-Takes-All Arbitration , 1997 .

[23]  Ah Chung Tsoi,et al.  Face recognition: a convolutional neural-network approach , 1997, IEEE Trans. Neural Networks.

[24]  Eric A. Vittoz,et al.  An integrated cortical layer for orientation enhancement , 1997 .

[25]  M. Arias-Estrada,et al.  Motion vision sensor architecture with asynchronous self-signaling pixels , 1997, Proceedings Fourth IEEE International Workshop on Computer Architecture for Machine Perception. CAMP'97.

[26]  W. D. Ross,et al.  Visual brain and visual perception: how does the cortex do perceptual grouping? , 1997, Trends in Neurosciences.

[27]  Gert Cauwenberghs,et al.  An analog VLSI chip with asynchronous interface for auditory feature extraction , 1997, Proceedings of 1997 IEEE International Symposium on Circuits and Systems. Circuits and Systems in the Information Age ISCAS '97.

[28]  Gert Cauwenberghs,et al.  An analog VLSI chip with asynchronous interface for auditory feature extraction , 1998 .

[29]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[30]  Wulfram Gerstner,et al.  Spike-Based Compared to Rate-Based Hebbian Learning , 1998, NIPS.

[31]  Kwabena Boahen,et al.  Communicating neuronal ensembles between neuromorphic chips , 1998 .

[32]  Claus Nebauer,et al.  Evaluation of convolutional neural networks for visual recognition , 1998, IEEE Trans. Neural Networks.

[33]  Kwabena Boahen,et al.  Retinomorphic chips that see quadruple images , 1999, Proceedings of the Seventh International Conference on Microelectronics for Neural, Fuzzy and Bio-Inspired Systems.

[34]  Stephen Grossberg,et al.  A neural network for enhancing boundaries and surfaces in synthetic aperture radar images , 1999, Neural Networks.

[35]  Jacques Gautrais,et al.  SpikeNET: A simulator for modeling large networks of integrate and fire neurons , 1999, Neurocomputing.

[36]  Andreas G. Andreou,et al.  AER image filtering architecture for vision-processing systems , 1999 .

[37]  Giacomo Indiveri,et al.  A reconfigurable neuromorphic VLSI multi-chip system applied to visual motion computation , 1999, Proceedings of the Seventh International Conference on Microelectronics for Neural, Fuzzy and Bio-Inspired Systems.

[38]  Kwabena Boahen,et al.  Point-to-point connectivity between neuromorphic chips using address events , 2000 .

[39]  Arnaud Delorme,et al.  Networks of integrate-and-fire neurons using Rank Order Coding B: Spike timing dependent plasticity and emergence of orientation selectivity , 2001, Neurocomputing.

[40]  Emery N. Brown,et al.  The Time-Rescaling Theorem and Its Application to Neural Spike Train Data Analysis , 2002, Neural Computation.

[41]  Beat Fasel,et al.  Robust face analysis using convolutional neural networks , 2002, Object recognition supported by user interaction for service robots.

[42]  Katsuhiko Mori,et al.  Convolutional spiking neural network model for robust face detection , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[43]  Charles M. Higgins,et al.  A biologically inspired modular VLSI system for visual measurement of self-motion , 2002 .

[44]  Shih-Chii Liu,et al.  Modeling Short-Term Synaptic Depression in Silicon , 2003, Neural Computation.

[45]  Saeed Shiry Ghidary,et al.  Convolutional Neural Networks for Image Processing: An Application in Robot Vision , 2003, Australian Conference on Artificial Intelligence.

[46]  E. Culurciello,et al.  A biomorphic digital image sensor , 2003, IEEE J. Solid State Circuits.

[47]  Philipp Häfliger,et al.  A multi-level static memory cell , 2003, ISCAS.

[48]  Pierre-Yves Burgi,et al.  A 128 /spl times/ 128 pixel 120 dB dynamic range vision sensor chip for image contrast and orientation extraction , 2003, 2003 IEEE International Solid-State Circuits Conference, 2003. Digest of Technical Papers. ISSCC..

[49]  Bertram E. Shi,et al.  An ON-OFF orientation selective address event representation image transceiver chip , 2004, IEEE Transactions on Circuits and Systems I: Regular Papers.

[50]  Kwabena Boahen.  A burst-mode word-serial address-event link-II: receiver design , 2004, IEEE Transactions on Circuits and Systems I: Regular Papers.

[51]  John G. Harris,et al.  A time-to-first spike CMOS imager , 2004, 2004 IEEE International Symposium on Circuits and Systems (IEEE Cat. No.04CH37512).

[52]  Philipp Häfliger,et al.  A time domain winner-take-all network of integrate-and-fire neurons , 2004, 2004 IEEE International Symposium on Circuits and Systems (IEEE Cat. No.04CH37512).

[53]  Giacomo Indiveri,et al.  An event-based VLSI network of integrate-and-fire neurons , 2004, 2004 IEEE International Symposium on Circuits and Systems (IEEE Cat. No.04CH37512).

[54]  Gert Cauwenberghs,et al.  Spatial acuity modulation of an address-event imager , 2004, Proceedings of the 2004 11th IEEE International Conference on Electronics, Circuits and Systems, 2004. ICECS 2004..

[55]  Kwabena Boahen,et al.  Optic nerve signals in a neuromorphic chip I: Outer and inner retina models , 2004, IEEE Transactions on Biomedical Engineering.

[56]  Kwabena Boahen.  A burst-mode word-serial address-event link-III: analysis and test results , 2004, IEEE Transactions on Circuits and Systems I: Regular Papers.

[57]  Kwabena Boahen,et al.  A burst-mode word-serial address-event link-I: transmitter design , 2004, IEEE Transactions on Circuits and Systems I: Regular Papers.

[58]  Amine Bermak,et al.  A low power CMOS imager based on time-to-first-spike encoding and fair AER , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[59]  L. Itti.  Author address: , 1999 .

[60]  Friedrich T. Sommer,et al.  Computing with inter-spike interval codes in networks of integrate and fire neurons , 2005, Neurocomputing.

[61]  Shih-Chii Liu,et al.  Spiking Inputs to a Winner-take-all Network , 2005, NIPS.

[62]  Alejandro Linares-Barranco,et al.  Two Hardware Implementations of the Exhaustive Synthetic AER Generation Method , 2005, IWANN.

[63]  Mehdi Azadmehr,et al.  A foveated AER imager chip [address event representation] , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[64]  Erhan Ozalevli,et al.  Reconfigurable biologically inspired visual motion systems using modular neuromorphic VLSI chips , 2005, IEEE Transactions on Circuits and Systems I: Regular Papers.

[65]  Wayne Luk,et al.  Have GPUs made FPGAs redundant in the field of video processing? , 2005, Proceedings. 2005 IEEE International Conference on Field-Programmable Technology, 2005..

[66]  Mehdi Azadmehr.  A foveated aer imager chip , 2005 .

[67]  Timothy K. Horiuchi,et al.  A VLSI model of the bat dorsal nucleus of the lateral lemniscus for azimuthal echolocation , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[68]  R. Etienne-Cummings,et al.  Temporal change threshold detection imager , 2005, ISSCC. 2005 IEEE International Digest of Technical Papers. Solid-State Circuits Conference, 2005..

[69]  Eugenio Culurciello,et al.  Event-based imaging with active illumination in sensor networks , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[70]  Bernabé Linares-Barranco,et al.  On algorithmic rate-coded AER generation , 2006, IEEE Transactions on Neural Networks.

[71]  Alejandro Linares-Barranco,et al.  AER tools for communications and debugging , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[72]  Shih-Chii Liu,et al.  Feature competition in a spike-based winner-take-all VLSI network , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[73]  T. Delbruck,et al.  A 128 128 120 dB 15 s Latency Asynchronous Temporal Contrast Vision Sensor , 2006 .

[74]  Tobi Delbrück,et al.  Fully programmable bias current generator with 24 bit resolution per bias , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[75]  Tobi Delbrück,et al.  A 128 X 128 120db 30mw asynchronous vision sensor that responds to relative intensity change , 2006, 2006 IEEE International Solid State Circuits Conference - Digest of Technical Papers.

[76]  L. Itti.  Quantitative modelling of perceptual salience at human eye position , 2006 .

[77]  Bernabé Linares-Barranco,et al.  A Neuromorphic Cortical-Layer Microchip for Spike-Based Event Processing Vision Systems , 2006, IEEE Transactions on Circuits and Systems I: Regular Papers.

[78]  R. Douglas,et al.  A multi-chip pulse-based neuromorphic infrastructure and its application to a cortical model of orientation selectivity , 2006 .

[79]  Tomaso Poggio,et al.  Learning a dictionary of shape-components in visual cortex: comparison with neurons, humans and machines , 2006 .

[80]  Tobi Delbrück,et al.  A 5 Meps $100 USB2.0 Address-Event Monitor-Sequencer Interface , 2007, 2007 IEEE International Symposium on Circuits and Systems.

[81]  Bernabé Linares-Barranco,et al.  Inter-spike-intervals analysis of AER Poisson-like generator hardware , 2007, Neurocomputing.

[82]  Philipp Häfliger.  Adaptive WTA With an Analog VLSI Neuromorphic Learning Chip , 2007, IEEE Transactions on Neural Networks.

[83]  Tobi Delbrück,et al.  A Multichip Pulse-Based Neuromorphic Infrastructure and Its Application to a Model of Orientation Selectivity , 2007, IEEE Transactions on Circuits and Systems I: Regular Papers.

[84]  Tobi Delbrück,et al.  Fast sensory motor control based on event-based hybrid neuromorphic-procedural system , 2007, 2007 IEEE International Symposium on Circuits and Systems.

[85]  Bernabé Linares-Barranco,et al.  A Spatial Contrast Retina With On-Chip Calibration for Neuromorphic Spike-Based AER Vision Systems , 2007, IEEE Transactions on Circuits and Systems I: Regular Papers.

[86]  André van Schaik,et al.  AER EAR: A Matched Silicon Cochlea Pair With Address Event Representation Interface , 2005, IEEE Transactions on Circuits and Systems I: Regular Papers.

[87]  Shih-Chii Liu,et al.  Quantifying Input and Output Spike Statistics of a Winner-Take-All Network in a Vision System , 2007, 2007 IEEE International Symposium on Circuits and Systems.

[88]  A. Civit,et al.  Address-event-based platform for bioinspired spiking systems , 2007, SPIE Microtechnologies.

[89]  Bernabé Linares-Barranco,et al.  High-speed character recognition system based on a complex hierarchical AER architecture , 2008, 2008 IEEE International Symposium on Circuits and Systems.

[90]  Tobi Delbrück,et al.  A 128$\times$ 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor , 2008, IEEE Journal of Solid-State Circuits.

[91]  Yingxue Wang,et al.  Quantification of a Spike-Based Winner-Take-All VLSI Network , 2008, IEEE Transactions on Circuits and Systems I: Regular Papers.

[92]  Tobias Delbrück,et al.  Frame-free dynamic digital vision , 2008 .

[93]  Bernabé Linares-Barranco,et al.  Fully digital AER convolution chip for vision processing , 2008, 2008 IEEE International Symposium on Circuits and Systems.

[94]  Bernabé Linares-Barranco,et al.  On Real-Time AER 2-D Convolutions Hardware for Neuromorphic Spike-Based Cortical Processing , 2008, IEEE Transactions on Neural Networks.

[95]  Shih-Chii Liu,et al.  Computation with Spikes in a Winner-Take-All Network , 2009, Neural Computation.

[96]  Peter Tino,et al.  IEEE Transactions on Neural Networks , 2009 .