Benchmarking Spike-Based Visual Recognition: A Dataset and Evaluation

Today, increasing attention is being paid to research into spike-based neural computation both to gain a better understanding of the brain and to explore biologically-inspired computation. Within this field, the primate visual pathway and its hierarchical organization have been extensively studied. Spiking Neural Networks (SNNs), inspired by the understanding of observed biological structure and function, have been successfully applied to visual recognition and classification tasks. In addition, implementations on neuromorphic hardware have enabled large-scale networks to run in (or even faster than) real time, making spike-based neural vision processing accessible on mobile robots. Neuromorphic sensors such as silicon retinas are able to feed such mobile systems with real-time visual stimuli. A new set of vision benchmarks for spike-based neural processing are now needed to measure progress quantitatively within this rapidly advancing field. We propose that a large dataset of spike-based visual stimuli is needed to provide meaningful comparisons between different systems, and a corresponding evaluation methodology is also required to measure the performance of SNN models and their hardware implementations. In this paper we first propose an initial NE (Neuromorphic Engineering) dataset based on standard computer vision benchmarksand that uses digits from the MNIST database. This dataset is compatible with the state of current research on spike-based image recognition. The corresponding spike trains are produced using a range of techniques: rate-based Poisson spike generation, rank order encoding, and recorded output from a silicon retina with both flashing and oscillating input stimuli. In addition, a complementary evaluation methodology is presented to assess both model-level and hardware-level performance. Finally, we demonstrate the use of the dataset and the evaluation methodology using two SNN models to validate the performance of the models and their hardware implementations. With this dataset we hope to (1) promote meaningful comparison between algorithms in the field of neural computation, (2) allow comparison with conventional image recognition methods, (3) provide an assessment of the state of the art in spike-based visual recognition, and (4) help researchers identify future directions and advance the field.

[1]  Steve B. Furber,et al.  Power analysis of large-scale, real-time neural networks on SpiNNaker , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[2]  Tobias Delbrück,et al.  Frame-free dynamic digital vision , 2008 .

[3]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[4]  Jongkil Park,et al.  A 65k-neuron 73-Mevents/s 22-pJ/event asynchronous micro-pipelined integrate-and-fire array transceiver , 2014, 2014 IEEE Biomedical Circuits and Systems Conference (BioCAS) Proceedings.

[5]  Shih-Chii Liu,et al.  Minitaur, an Event-Driven FPGA-Based Spiking Network Accelerator , 2014, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[6]  Bernabé Linares-Barranco,et al.  Feedforward Categorization on AER Motion Events Using Cortex-Like Features in a Spiking Neural Network , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[7]  Damien Querlioz,et al.  Extraction of temporally correlated features from dynamic vision sensors with spike-timing-dependent plasticity , 2012, Neural Networks.

[8]  B J Melloni,et al.  How the retina works. , 1971, American Family Physician.

[9]  A. L. Yarbus.  Eye Movements During Perception of Complex Objects , 1967 .

[10]  Walter Senn,et al.  Learning Real-World Stimuli in a Neural Network with Spike-Driven Synaptic Dynamics , 2007, Neural Computation.

[11]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[12]  S. Kosslyn,et al.  Findings and current opinion in cognitive neuroscience , 1998 .

[13]  Bernabé Linares-Barranco,et al.  A 128$\,\times$ 128 1.5% Contrast Sensitivity 0.9% FPN 3 µs Latency 4 mW Asynchronous Frame-Free Dynamic Vision Sensor Using Transimpedance Preamplifiers , 2013, IEEE Journal of Solid-State Circuits.

[14]  Romain Brette,et al.  Brian: A Simulator for Spiking Neural Networks in Python , 2008, Frontiers Neuroinformatics.

[15]  Rufin van Rullen,et al.  Rate Coding Versus Temporal Order Coding: What the Retinal Ganglion Cells Tell the Visual Cortex , 2001, Neural Computation.

[16]  Arnaud Delorme,et al.  Face identification using one spike per neuron: resistance to image degradations , 2001, Neural Networks.

[17]  Olivier Temam,et al.  Hardware spiking neurons design: Analog or digital? , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[18]  Steve B. Furber,et al.  Evaluating rank-order code performance using a biologically-derived retinal model , 2009, 2009 International Joint Conference on Neural Networks.

[19]  Shahin Nazarian,et al.  Thermal Modeling, Analysis, and Management in VLSI Circuits: Principles and Methods , 2006, Proceedings of the IEEE.

[20]  Steve B. Furber,et al.  The SpiNNaker Project , 2014, Proceedings of the IEEE.

[21]  Steve B. Furber,et al.  Live demonstration: Handwritten digit recognition using spiking deep belief networks on SpiNNaker , 2015, 2015 IEEE International Symposium on Circuits and Systems (ISCAS).

[22]  Romain Brette,et al.  Neuroinformatics Original Research Article Brian: a Simulator for Spiking Neural Networks in Python , 2022 .

[23]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[24]  Tobi Delbruck,et al.  A Dynamic Vision Sensor With 1% Temporal Contrast Sensitivity and In-Pixel Asynchronous Delta Modulator for Event Encoding , 2015, IEEE Journal of Solid-State Circuits.

[25]  Marc-Oliver Gewaltig,et al.  NEST (NEural Simulation Tool) , 2007, Scholarpedia.

[26]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[27]  Steve B. Furber,et al.  Neural Systems Engineering , 2008, Computational Intelligence: A Compendium.

[28]  Steve Furber,et al.  Power-efficient simulation of detailed cortical microcircuits on SpiNNaker , 2012, Journal of Neuroscience Methods.

[29]  James J. DiCarlo,et al.  How Does the Brain Solve Visual Object Recognition? , 2012, Neuron.

[30]  Matthew Cook,et al.  Unsupervised learning of digit recognition using spike-timing-dependent plasticity , 2015, Front. Comput. Neurosci..

[31]  Jiebo Luo,et al.  Recognizing realistic actions from videos “in the wild” , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  S. Thorpe,et al.  Surfing a spike wave down the ventral stream , 2002, Vision Research.

[33]  Johannes Schemmel,et al.  A wafer-scale neuromorphic hardware system for large-scale neural modeling , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[34]  Robert Legenstein,et al.  A compound memristive synapse model for statistical learning through STDP in spiking neural networks , 2014, Front. Neurosci..

[35]  Rodrigo Alvarez-Icaza,et al.  Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural Simulations , 2014, Proceedings of the IEEE.

[36]  T. Poggio,et al.  Hierarchical models of object recognition in cortex , 1999, Nature Neuroscience.

[37]  Trevor Bekolay,et al.  A Large-Scale Model of the Functioning Brain , 2012, Science.

[38]  Steve B. Furber,et al.  Accuracy and Efficiency in Fixed-Point Neural ODE Solvers , 2015, Neural Computation.

[39]  Pierre Yger,et al.  PyNN: A Common Interface for Neuronal Network Simulators , 2008, Front. Neuroinform..

[40]  Deepak Khosla,et al.  Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition , 2014, International Journal of Computer Vision.

[41]  Katsuhiko Mori,et al.  Convolutional spiking neural network model for robust face detection , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[42]  Gert Cauwenberghs,et al.  Neuromorphic Silicon Neuron Circuits , 2011, Front. Neurosci.

[43]  PoggioTomaso,et al.  Robust Object Recognition with Cortex-Like Mechanisms , 2007 .

[44]  Steve B. Furber,et al.  Robustness of spiking Deep Belief Networks to noise and reduced bit precision of neuro-inspired hardware platforms , 2015, Front. Neurosci..

[45]  Gregory Cohen,et al.  Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades , 2015, Front. Neurosci..

[46]  Tobi Delbruck,et al.  Real-time classification and sensor fusion with a spiking deep belief network , 2013, Front. Neurosci..

[47]  Tobi Delbrück,et al.  Retinomorphic Event-Based Vision Sensors: Bioinspired Cameras With Spiking Output , 2014, Proceedings of the IEEE.

[48]  Matthew Cook,et al.  Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[49]  Jiebo Luo,et al.  Recognizing realistic actions from videos , 2009, CVPR.

[50]  Y. LeCun,et al.  Learning methods for generic object recognition with invariance to pose and lighting , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[51]  S. Thorpe,et al.  Rapid categorization of natural images by rhesus monkeys , 1998, Neuroreport.

[52]  Siddharth Joshi,et al.  Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines , 2015, Front. Neurosci..

[53]  Gert Cauwenberghs,et al.  Event-driven contrastive divergence for spiking neuromorphic systems , 2013, Front. Neurosci..

[54]  S. Joshi,et al.  65k-neuron integrate-and-fire array transceiver with address-event reconfigurable synaptic routing , 2012, 2012 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[55]  Garrick Orchard,et al.  Benchmarking neuromorphic vision: lessons learnt from computer vision , 2015, Front. Neurosci..

[56]  H. Kolb.  How the Retina Works , 2003, American Scientist.

[57]  Wolfgang Maass,et al.  Bayesian Computation Emerges in Generic Cortical Microcircuits through Spike-Timing-Dependent Plasticity , 2013, PLoS Comput. Biol..

[58]  Nikil D. Dutt,et al.  Categorization and decision-making in a neurobiologically plausible spiking network using a STDP-like learning rule , 2013, Neural Networks.

[59]  P. Fldik,et al.  The Speed of Sight , 2001, Journal of Cognitive Neuroscience.

[60]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[61]  Guosheng Yang,et al.  A Spiking Neural Network Based Cortex-Like Mechanism and Application to Facial Expression Recognition , 2012, Comput. Intell. Neurosci..

[62]  Bernabé Linares-Barranco,et al.  Compact low-power calibration mini-DACs for neural arrays with programmable weights , 2003, IEEE Trans. Neural Networks.

[63]  Steve Furber,et al.  Real-Time Recognition of Dynamic Hand Postures on a Neuromorphic System , 2015 .

[64]  Bernabé Linares-Barranco,et al.  An Event-Driven Multi-Kernel Convolution Processor Module for Event-Driven Vision Sensors , 2012, IEEE Journal of Solid-State Circuits.

[65]  Florian Jug,et al.  Spiking networks and their rate-based equivalents : does it make sense to use Siegert neurons ? , 2011 .

[66]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[67]  Gaël Varoquaux,et al.  The NumPy Array: A Structure for Efficient Numerical Computation , 2011, Computing in Science & Engineering.

[68]  Steve B. Furber,et al.  Scalable energy-efficient, low-latency implementations of trained spiking Deep Belief Networks on SpiNNaker , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[69]  Pierre Kornprobst,et al.  A novel bio-inspired static image compression scheme for noisy data transmission over low-bandwidth channels , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[70]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[71]  Misha Anne Mahowald,et al.  VLSI analogs of neuronal visual processing: a synthesis of form and function , 1992 .

[72]  Ronen Basri,et al.  Actions as space-time shapes , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[73]  Moritz Helias,et al.  Statistical properties of superimposed stationary spike trains , 2011, Journal of Computational Neuroscience.