A Neuromorphic Architecture for Object Recognition and Motion Anticipation Using Burst-STDP

In this work we investigate the possibilities offered by a minimal framework of artificial spiking neurons to be deployed in silico. Here we introduce a hierarchical network architecture of spiking neurons which learns to recognize moving objects in a visual environment and determine the correct motor output for each object. These tasks are learned through both supervised and unsupervised spike timing dependent plasticity (STDP). STDP is responsible for the strengthening (or weakening) of synapses in relation to pre- and post-synaptic spike times and has been described as a Hebbian paradigm taking place both in vitro and in vivo. We utilize a variation of STDP learning, called burst-STDP, which is based on the notion that, since spikes are expensive in terms of energy consumption, then strong bursting activity carries more information than single (sparse) spikes. Furthermore, this learning algorithm takes advantage of homeostatic renormalization, which has been hypothesized to promote memory consolidation during NREM sleep. Using this learning rule, we design a spiking neural network architecture capable of object recognition, motion detection, attention towards important objects, and motor control outputs. We demonstrate the abilities of our design in a simple environment with distractor objects, multiple objects moving concurrently, and in the presence of noise. Most importantly, we show how this neural network is capable of performing these tasks using a simple leaky-integrate-and-fire (LIF) neuron model with binary synapses, making it fully compatible with state-of-the-art digital neuromorphic hardware designs. As such, the building blocks and learning rules presented in this paper appear promising for scalable fully neuromorphic systems to be implemented in hardware chips.

[1]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[2]  D. Hubel,et al.  Receptive fields and functional architecture of monkey striate cortex , 1968, The Journal of physiology.

[3]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[4]  M. Goodale,et al.  Separate visual pathways for perception and action , 1992, Trends in Neurosciences.

[5]  G. Edelman,et al.  Reentry and the problem of integrating multiple cortical areas: simulation of dynamic integration in the visual system. , 1992, Cerebral cortex.

[6]  T. Sejnowski,et al.  A selection model for motion processing in area MT of primates , 1995, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[7]  John K. Tsotsos,et al.  Modeling Visual Attention via Selective Tuning , 1995, Artif. Intell..

[8]  Denis Fize,et al.  Speed of processing in the human visual system , 1996, Nature.

[9]  T J Sejnowski,et al.  A Model for Encoding Multiple Object Motions and Self-Motion in Area MST of Primate Visual Cortex , 1998, The Journal of Neuroscience.

[10]  S. Edelman,et al.  Human Brain Mapping 6:316–328(1998) � A Sequence of Object-Processing Stages Revealed by fMRI in the Human Occipital Lobe , 2022 .

[11]  Eero P. Simoncelli,et al.  A model of neuronal responses in visual area MT , 1998, Vision Research.

[12]  Victor A. F. Lamme,et al.  Feedforward, horizontal, and feedback processing in the visual cortex , 1998, Current Opinion in Neurobiology.

[13]  J A Perrone,et al.  Emulating the Visual Receptive-Field Properties of MST Neurons with a Template Model of Heading Estimation , 1998, The Journal of Neuroscience.

[14]  G. Bi,et al.  Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type , 1998, The Journal of Neuroscience.

[15]  L M Vaina,et al.  Computational modelling of optic flow selectivity in MSTd neurons. , 1998, Network.

[16]  P. Achermann,et al.  Sleep homeostasis and models of sleep regulation. , 1999, Journal of biological rhythms.

[17]  T. Poggio,et al.  Hierarchical models of object recognition in cortex , 1999, Nature Neuroscience.

[18]  Rajesh P. N. Rao,et al.  Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. , 1999 .

[19]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[20]  L. Abbott,et al.  Synaptic plasticity: taming the beast , 2000, Nature Neuroscience.

[21]  L. Abbott,et al.  Competitive Hebbian learning through spike-timing-dependent synaptic plasticity , 2000, Nature Neuroscience.

[22]  G. Tononi,et al.  Differential Expression of Plasticity-Related Genes in Waking and Sleep and Their Regulation by the Noradrenergic System , 2000, The Journal of Neuroscience.

[23]  Christopher C. Pack,et al.  A Neural Model of Smooth Pursuit Control and Motion Perception by Cortical Area MST , 2001, Journal of Cognitive Neuroscience.

[24]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[25]  Stephen Grossberg,et al.  Neural dynamics of motion integration and segmentation within and across apertures , 2001, Vision Research.

[26]  S. Laughlin,et al.  An Energy Budget for Signaling in the Grey Matter of the Brain , 2001, Journal of cerebral blood flow and metabolism : official journal of the International Society of Cerebral Blood Flow and Metabolism.

[27]  L. Cooper,et al.  A biophysical model of bidirectional synaptic plasticity: Dependence on AMPA and NMDA receptors , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[28]  Martin A. Giese,et al.  Biophysiologically Plausible Implementations of the Maximum Operation , 2002, Neural Computation.

[29]  Gastone C. Castellani,et al.  Converging evidence for a simplified biophysical model of synaptic plasticity , 2002, Biological Cybernetics.

[30]  Christof Koch,et al.  Attentional Selection for Object Recognition - A Gentle Way , 2002, Biologically Motivated Computer Vision.

[31]  T. Meese,et al.  Spiral mechanisms are required to account for summation of complex motion components , 2002, Vision Research.

[32]  Dirk B. Walther,et al.  Attentional Selection for Object Recognition – a Gentle , 2002 .

[33]  Olaf Sporns,et al.  Neuromodulation and plasticity in an autonomous robot , 2002, Neural Networks.

[34]  G. Tononi,et al.  Sleep and synaptic homeostasis: a hypothesis , 2003, Brain Research Bulletin.

[35]  T. Poggio,et al.  Cognitive neuroscience: Neural mechanisms for the recognition of biological movements , 2003, Nature Reviews Neuroscience.

[36]  David B. Grayden,et al.  Spike-Timing-Dependent Plasticity: The Relationship to Rate-Based Learning for Models with Weight Dynamics Determined by a Stable Fixed Point , 2004, Neural Computation.

[37]  Giacomo Indiveri,et al.  A VLSI reconfigurable network of integrate-and-fire neurons with spike-based learning synapses , 2004, ESANN.

[38]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[39]  Y. Dan,et al.  Spike Timing-Dependent Plasticity of Neural Circuits , 2004, Neuron.

[40]  G. Tononi,et al.  Extensive and Divergent Effects of Sleep and Wakefulness on Brain Gene Expression , 2004, Neuron.

[41]  Kunihiko Fukushima,et al.  A neural network model for selective attention in visual pattern recognition , 1986, Biological Cybernetics.

[42]  Eugene M. Izhikevich,et al.  Which model to use for cortical spiking neurons? , 2004, IEEE Transactions on Neural Networks.

[43]  John K. Tsotsos,et al.  Attending to visual motion , 2005, Comput. Vis. Image Underst..

[44]  Pieter R. Roelfsema,et al.  Attention-Gated Reinforcement Learning of Internal Representations for Classification , 2005, Neural Computation.

[45]  Johannes Schemmel,et al.  Training convolutional networks of threshold neurons suited for low-power hardware implementation , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[46]  G. Tononi,et al.  Sleep function and synaptic homeostasis. , 2006, Sleep medicine reviews.

[47]  Simon J. Thorpe,et al.  Ultra-rapid object detection with saccadic eye movements: Visual processing speed revisited , 2006, Vision Research.

[48]  Patrick Le Callet,et al.  A coherent computational approach to model bottom-up visual attention , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  David G. Lowe,et al.  Multiclass Object Recognition with Sparse, Localized Features , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[50]  Thomas Serre,et al.  A Biologically Inspired System for Action Recognition , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[51]  R. Etienne-Cummings,et al.  Spike-Based MAX Networks for Nonlinear Pooling in Hierarchical Vision Processing , 2007, 2007 IEEE Biomedical Circuits and Systems Conference.

[52]  Thomas Serre,et al.  A feedforward architecture accounts for rapid categorization , 2007, Proceedings of the National Academy of Sciences.

[53]  Walter Senn,et al.  Learning Real-World Stimuli in a Neural Network with Spike-Driven Synaptic Dynamics , 2007, Neural Computation.

[54]  Laurent Itti,et al.  Beyond bottom-up: Incorporating task-dependent influences into a computational model of spatial attention , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[55]  A. Kirkwood,et al.  Neuromodulators Control the Polarity of Spike-Timing-Dependent Synaptic Plasticity , 2007, Neuron.

[56]  Laurent Itti,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Rapid Biologically-inspired Scene Classification Using Features Shared with Visual Attention , 2022 .

[57]  Thomas P. Trappenberg,et al.  Computational consequences of experimentally derived spike-time and weight dependent plasticity rules , 2007, Biological Cybernetics.

[58]  Xiao‐Bing Gao,et al.  Prolonged wakefulness induces experience-dependent synaptic plasticity in mouse hypocretin/orexin neurons. , 2007, The Journal of clinical investigation.

[59]  Timothée Masquelier,et al.  Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity , 2007, PLoS Comput. Biol..

[60]  Nicolas Pinto,et al.  Why is Real-World Visual Object Recognition Hard? , 2008, PLoS Comput. Biol..

[61]  Johannes Schemmel,et al.  Realizing biological spiking network models in a configurable wafer-scale hardware system , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[62]  Donghyun Kim,et al.  A 125GOPS 583mW Network-on-Chip Based Parallel Processor with Bio-inspired Visual-Attention Engine , 2008, 2008 IEEE International Solid-State Circuits Conference - Digest of Technical Papers.

[63]  G. Tononi,et al.  Molecular and electrophysiological evidence for net synaptic potentiation in wake and depression in sleep , 2008, Nature Neuroscience.

[64]  Giorgio F. Gilestro,et al.  Widespread Changes in Synaptic Markers as a Function of Sleep and Wakefulness in Drosophila , 2009, Science.

[65]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[66]  G. Tononi,et al.  Cortical Firing and Sleep Homeostasis , 2009, Neuron.

[67]  Joo-Young Kim,et al.  A 125 GOPS 583 mW Network-on-Chip Based Parallel Processor With Bio-Inspired Visual Attention Engine , 2009, IEEE Journal of Solid-State Circuits.

[68]  Laurent Itti,et al.  Biologically Inspired Mobile Robot Vision Localization , 2009, IEEE Transactions on Robotics.

[69]  W. Gan,et al.  Dendritic spine dynamics. , 2009, Annual review of physiology.

[70]  Johannes Schemmel,et al.  Neuroinformatics Original Research Article Establishing a Novel Modeling Tool: a Python-based Interface for a Neuromorphic Hardware System , 2022 .

[71]  Bernabé Linares-Barranco,et al.  Fast Vision Through Frameless Event-Based Sensing and Convolutional Processing: Application to Texture Recognition , 2010, IEEE Transactions on Neural Networks.

[72]  G. Tononi,et al.  Sleep and synaptic renormalization: a computational study. , 2010, Journal of neurophysiology.

[73]  T. Sejnowski,et al.  Metabolic cost as a unifying principle governing neuronal biophysics , 2010, Proceedings of the National Academy of Sciences.

[74]  Dacheng Tao,et al.  Biologically Inspired Feature Manifold for Scene Classification , 2010, IEEE Transactions on Image Processing.

[75]  Johannes Schemmel,et al.  A wafer-scale neuromorphic hardware system for large-scale neural modeling , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[76]  Heiko Neumann,et al.  Interactions of motion and form in visual cortex – A neural model , 2008, Journal of Physiology-Paris.

[77]  G. Tononi,et al.  Direct Evidence for Wake-Related Increases and Sleep-Related Decreases in Synaptic Strength in Rodent Cortex , 2010, The Journal of Neuroscience.

[78]  Timothée Masquelier,et al.  Relative spike time coding and STDP-based orientation selectivity in the early visual system in natural continuous and saccadic vision: a computational model , 2011, Journal of Computational Neuroscience.

[79]  Dharmendra S. Modha,et al.  A digital neurosynaptic core using embedded crossbar memory with 45pJ per spike in 45nm , 2011, 2011 IEEE Custom Integrated Circuits Conference (CICC).

[80]  John K. Tsotsos A Computational Perspective on Visual Attention , 2011 .

[81]  Yong Liu,et al.  A 45nm CMOS neuromorphic chip with a scalable architecture for learning in networks of spiking neurons , 2011, 2011 IEEE Custom Integrated Circuits Conference (CICC).

[82]  Giulio Tononi,et al.  What can neurons do for their brain? Communicate selectivity with bursts , 2013, Theory in Biosciences.