Unsupervised AER Object Recognition Based on Multiscale Spatio-Temporal Features and Spiking Neurons

This article proposes an unsupervised address event representation (AER) object recognition approach. The proposed approach consists of a novel multiscale spatio-temporal feature (MuST) representation of input AER events and a spiking neural network (SNN) using spike-timing-dependent plasticity (STDP) for object recognition with MuST. MuST extracts the features contained in both the spatial and temporal information of AER event flow, and forms an informative and compact feature spike representation. We show not only how MuST exploits spikes to convey information more effectively, but also how it benefits the recognition using SNN. The recognition process is performed in an unsupervised manner, which does not need to specify the desired status of every single neuron of SNN, and thus can be flexibly applied in real-world recognition tasks. The experiments are performed on five AER datasets including a new one named GESTURE-DVS. Extensive experimental results show the effectiveness and advantages of the proposed approach.

[1]  Nicolas Brunel,et al.  Sensory neural codes using multiplexed temporal scales , 2010, Trends in Neurosciences.

[2]  Haizhou Li,et al.  Rapid Feedforward Computation by Temporal Encoding and Learning With Spiking Neurons , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[3]  Rong Xiao,et al.  An Event-Driven Computational System with Spiking Neurons for Object Recognition , 2017, ICONIP.

[4]  J. Pfister,et al.  A triplet spike-timing–dependent plasticity model generalizes the Bienenstock–Cooper–Munro rule to higher-order spatiotemporal correlations , 2011, Proceedings of the National Academy of Sciences.

[5]  Tobi Delbrück,et al.  A 128$\times$ 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor , 2008, IEEE Journal of Solid-State Circuits.

[6]  Gang Pan,et al.  Spiking Deep Residual Network , 2018, ArXiv.

[7]  T. Delbruck,et al.  > Replace This Line with Your Paper Identification Number (double-click Here to Edit) < 1 , 2022 .

[8]  Matthew Cook,et al.  Unsupervised learning of digit recognition using spike-timing-dependent plasticity , 2015, Front. Comput. Neurosci..

[9]  Giacomo Indiveri,et al.  Memory and Information Processing in Neuromorphic Systems , 2015, Proceedings of the IEEE.

[10]  Zhaohui Wu,et al.  Cyborg Intelligence: Towards Bio-Machine Intelligent Systems , 2014, IEEE Intelligent Systems.

[11]  Bernabé Linares-Barranco,et al.  A 3.6 $\mu$ s Latency Asynchronous Frame-Free Event-Driven Dynamic-Vision-Sensor , 2011, IEEE Journal of Solid-State Circuits.

[12]  Kaushik Roy,et al.  Unsupervised regenerative learning of hierarchical features in Spiking Deep Networks for object recognition , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[13]  Garrick Orchard,et al.  HOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Bo Zhao,et al.  Bag of Events: An Efficient Probability-Based Feature Extraction Method for AER Image Sensors , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[15]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[16]  Harry G. Barrow,et al.  The Role of Weight Normalization in Competitive Learning , 1994, Neural Computation.

[17]  D. Linden,et al.  The other side of the engram: experience-driven changes in neuronal intrinsic excitability , 2003, Nature Reviews Neuroscience.

[18]  Chun-I Yeh,et al.  Temporal precision in the neural code and the timescales of natural vision , 2007, Nature.

[19]  Tobi Delbruck,et al.  A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor , 2014, IEEE Journal of Solid-State Circuits.

[20]  Daniel Matolin,et al.  A QVGA 143 dB Dynamic Range Frame-Free PWM Image Sensor With Lossless Pixel-Level Video Compression and Time-Domain CDS , 2011, IEEE Journal of Solid-State Circuits.

[21]  Romain Brette,et al.  The Brian Simulator , 2009, Front. Neurosci..

[22]  Yi Zeng,et al.  A Plasticity-Centric Approach to Train the Non-Differential Spiking Neural Networks , 2018, AAAI.

[23]  Gregory Cohen,et al.  Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades , 2015, Front. Neurosci..

[24]  Bernabé Linares-Barranco,et al.  Feedforward Categorization on AER Motion Events Using Cortex-Like Features in a Spiking Neural Network , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[25]  Arindam Basu,et al.  Unsupervised learning of event-based image recordings using spike-timing-dependent plasticity , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[26]  G. Bi,et al.  Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type , 1998, The Journal of Neuroscience.

[27]  Don Monroe,et al.  Neuromorphic computing gets ready for the (really) big time , 2014, CACM.

[28]  Thomas Serre,et al.  A feedforward architecture accounts for rapid categorization , 2007, Proceedings of the National Academy of Sciences.

[29]  Thomas Serre,et al.  Robust Object Recognition with Cortex-Like Mechanisms , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Zhaohui Wu,et al.  Cyborg Intelligence: Recent Progress and Future Directions , 2016, IEEE Intelligent Systems.

[31]  Shigang Yue,et al.  Fast unsupervised learning for visual pattern recognition using spike timing dependent plasticity , 2017, Neurocomputing.

[32]  Bernabé Linares-Barranco,et al.  Mapping from Frame-Driven to Frame-Free Event-Driven Vision Systems by Low-Rate Rate Coding and Coincidence Processing--Application to Feedforward ConvNets , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  W. Gerstner,et al.  Triplets of Spikes in a Model of Spike Timing-Dependent Plasticity , 2006, The Journal of Neuroscience.

[34]  Haim Sompolinsky,et al.  Learning Precisely Timed Spikes , 2014, Neuron.

[35]  Haizhou Li,et al.  How the Brain Formulates Memory: A Spatio-Temporal Model Research Frontier , 2016, IEEE Computational Intelligence Magazine.

[36]  Nitish V. Thakor,et al.  HFirst: A Temporal Approach to Object Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Eugenio Culurciello,et al.  Efficient Feedforward Categorization of Objects and Human Postures with Address-Event Image Sensors , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Kay Chen Tan,et al.  Sparse Temporal Encoding of Visual Features for Robust Object Recognition by Spiking Neurons , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[39]  Zhaohui Wu,et al.  Jointly Learning Network Connections and Link Weights in Spiking Neural Networks , 2018, IJCAI.