Spiking Neural Networks for Inference and Learning: A Memristor-based Design Perspective

Author(s): Fouda, ME; Kurdahi, F; Eltawil, A; Neftci, E | Abstract: On metrics of density and power efficiency, neuromorphic technologies have the potential to surpass mainstream computing technologies in tasks where real-time functionality, adaptability, and autonomy are essential. While algorithmic advances in neuromorphic computing are proceeding successfully, the potential of memristors to improve neuromorphic computing have not yet born fruit, primarily because they are often used as a drop-in replacement to conventional memory. However, interdisciplinary approaches anchored in machine learning theory suggest that multifactor plasticity rules matching neural and synaptic dynamics to the device capabilities can take better advantage of memristor dynamics and its stochasticity. Furthermore, such plasticity rules generally show much higher performance than that of classical Spike Time Dependent Plasticity (STDP) rules. This chapter reviews the recent development in learning with spiking neural network models and their possible implementation with memristor-based hardware.

[1]  Gert Cauwenberghs,et al.  Neural and Synaptic Array Transceiver: A Brain-Inspired Computing Framework for Embedded Learning , 2017, Front. Neurosci..

[2]  Shimeng Yu,et al.  Resistive Memory-Based Analog Synapse: The Pursuit for Linear and Symmetric Weight Update , 2018, IEEE Nanotechnology Magazine.

[3]  Wulfram Gerstner,et al.  Predicting spike timing of neocortical pyramidal neurons by simple threshold models , 2006, Journal of Computational Neuroscience.

[4]  Gayle M. Wittenberg,et al.  Spike Timing Dependent Plasticity: A Consequence of More Fundamental Learning Rules , 2010, Front. Comput. Neurosci..

[5]  Qing Wu,et al.  Efficient and self-adaptive in-situ learning in multilayer memristor neural networks , 2018, Nature Communications.

[6]  Fabien Alibart,et al.  Plasticity in memristive devices for spiking neural networks , 2015, Front. Neurosci..

[7]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[8]  Rubén Moreno-Bote,et al.  Poisson-Like Spiking in Circuits with Probabilistic Synapses , 2014, PLoS Comput. Biol..

[9]  Ran El-Yaniv,et al.  Binarized Neural Networks , 2016, NIPS.

[10]  Xiaoyu Sun,et al.  Characterizing Endurance Degradation of Incremental Switching in Analog RRAM for Neuromorphic Systems , 2018, 2018 IEEE International Electron Devices Meeting (IEDM).

[11]  Somnath Paul,et al.  Event-driven random backpropagation: Enabling neuromorphic deep learning machines , 2017, 2017 IEEE International Symposium on Circuits and Systems (ISCAS).

[12]  Siddharth Joshi,et al.  Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines , 2015, Front. Neurosci..

[13]  Emre Neftci.  Stochastic synapses as resource for efficient deep learning machines , 2017, 2017 IEEE International Electron Devices Meeting (IEDM).

[14]  Alex Graves,et al.  Decoupled Neural Interfaces using Synthetic Gradients , 2016, ICML.

[15]  Xiaohui Xie,et al.  Equivalence of Backpropagation and Contrastive Hebbian Learning in a Layered Network , 2003, Neural Computation.

[16]  Daniele Ielmini,et al.  Brain-inspired computing with resistive switching memory (RRAM): Devices, synapses and neural networks , 2018 .

[17]  A. Faisal,et al.  Noise in the nervous system , 2008, Nature Reviews Neuroscience.

[18]  Garrick Orchard,et al.  SLAYER: Spike Layer Error Reassignment in Time , 2018, NeurIPS.

[19]  Luca Benini,et al.  YodaNN: An Ultra-Low Power Convolutional Neural Network Accelerator Based on Binary Weights , 2016, 2016 IEEE Computer Society Annual Symposium on VLSI (ISVLSI).

[20]  Hong Wang,et al.  Loihi: A Neuromorphic Manycore Processor with On-Chip Learning , 2018, IEEE Micro.

[21]  Wulfram Gerstner,et al.  SPIKING NEURON MODELS Single Neurons , Populations , Plasticity , 2002 .

[22]  G. Ghibaudo,et al.  Understanding RRAM endurance, retention and window margin trade-off using experimental results and simulations , 2016, 2016 IEEE International Electron Devices Meeting (IEDM).

[23]  W. Senn,et al.  Learning by the Dendritic Prediction of Somatic Spiking , 2014, Neuron.

[24]  H. Sompolinsky,et al.  The tempotron: a neuron that learns spike timing–based decisions , 2006, Nature Neuroscience.

[25]  裕幸 飯田,et al.  International Technology Roadmap for Semiconductors 2003の要求清浄度について － シリコンウエハ表面と雰囲気環境に要求される清浄度, 分析方法の現状について － , 2004 .

[26]  Chiara Bartolozzi,et al.  Implementing homeostatic plasticity in VLSI networks of spiking neurons , 2008, 2008 15th IEEE International Conference on Electronics, Circuits and Systems.

[27]  Somnath Paul,et al.  Event-Driven Random Back-Propagation: Enabling Neuromorphic Deep Learning Machines , 2016, Front. Neurosci..

[28]  Yoshua Bengio,et al.  Low precision arithmetic for deep learning , 2014, ICLR.

[29]  Bipin Rajendran,et al.  NormAD - Normalized Approximate Descent based supervised learning rule for spiking neurons , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[30]  Emre O. Neftci,et al.  Data and Power Efficient Intelligence with Neuromorphic Learning Machines , 2018, iScience.

[31]  Hyunsang Hwang,et al.  TiOx-Based RRAM Synapse With 64-Levels of Conductance and Symmetric Conductance Change by Adopting a Hybrid Pulse Scheme for Neuromorphic Computing , 2016, IEEE Electron Device Letters.

[32]  Robert A. Legenstein,et al.  Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets , 2019, ArXiv.

[33]  Ahmed M. Eltawil,et al.  Overcoming Crossbar Nonidealities in Binary Neural Networks Through Learning , 2018, 2018 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH).

[34]  Chiara Bartolozzi,et al.  Synaptic Dynamics in Analog VLSI , 2007, Neural Computation.

[35]  Rodrigo Alvarez-Icaza,et al.  Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural Simulations , 2014, Proceedings of the IEEE.

[36]  Jongkil Park,et al.  A 65k-neuron 73-Mevents/s 22-pJ/event asynchronous micro-pipelined integrate-and-fire array transceiver , 2014, 2014 IEEE Biomedical Circuits and Systems Conference (BioCAS) Proceedings.

[37]  Surya Ganguli,et al.  SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks , 2017, Neural Computation.

[38]  C. Bartolozzi,et al.  Silicon synapse implements multiple neural computational primitives , 2008 .

[39]  Travis Bartley,et al.  Contrastive Hebbian Learning with Random Feedback Weights , 2018, Neural Networks.

[40]  Steve B. Furber,et al.  The SpiNNaker Project , 2014, Proceedings of the IEEE.

[41]  Igor Carron,et al.  XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks , 2016 .

[42]  Carver A. Mead,et al.  Neuromorphic electronic systems , 1990, Proc. IEEE.

[43]  Colin J. Akerman,et al.  Random synaptic feedback weights support error backpropagation for deep learning , 2016, Nature Communications.

[44]  Giacomo Indiveri,et al.  A VLSI spike-driven dynamic synapse which learns only when necessary , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[45]  G. Bi,et al.  Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type , 1998, The Journal of Neuroscience.

[46]  Kaushik Roy,et al.  Rx-Caffe: Framework for evaluating and training Deep Neural Networks on Resistive Crossbars , 2018, ArXiv.

[47]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[48]  Shimeng Yu,et al.  Stochastic learning in oxide binary synaptic device for neuromorphic computing , 2013, Front. Neurosci..

[49]  Emre Neftci,et al.  Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks , 2019, IEEE Signal Processing Magazine.

[50]  Taro Toyoizumi,et al.  A Local Learning Rule for Independent Component Analysis , 2016, Scientific Reports.

[51]  S. Jeannot,et al.  Endurance/Retention Trade Off in HfOx and TaOx Based RRAM , 2016, 2016 IEEE 8th International Memory Workshop (IMW).

[52]  G. Cauwenberghs,et al.  Memristor-based neural networks: Synaptic versus neuronal stochasticity , 2016 .

[53]  Gert Cauwenberghs,et al.  Neuromorphic Silicon Neuron Circuits , 2011, Front. Neurosci.

[54]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[55]  Gert Cauwenberghs,et al.  Inherently stochastic spiking neurons for probabilistic neural computation , 2015, 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER).

[56]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[57]  Johannes Schemmel,et al.  A wafer-scale neuromorphic hardware system for large-scale neural modeling , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[58]  Terrence J. Sejnowski,et al.  Gradient Descent for Spiking Neural Networks , 2017, NeurIPS.

[59]  Jinseok Kim,et al.  Deep Neural Network Optimized to Resistive Memory with Nonlinear Current-Voltage Characteristics , 2017, ACM J. Emerg. Technol. Comput. Syst..

[60]  Johannes Schemmel,et al.  Wafer-scale integration of analog neural networks , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[61]  Jeanette Hellgren Kotaleski,et al.  GABAergic Circuits Control Spike-Timing-Dependent Plasticity , 2013, The Journal of Neuroscience.

[62]  Sander M. Bohte,et al.  Fast and Efficient Asynchronous Neural Computation with Adapting Spiking Neural Networks , 2016, ArXiv.

[63]  Sander M. Bohte,et al.  SpikeProp: backpropagation for networks of spiking neurons , 2000, ESANN.

[64]  Surya Ganguli,et al.  A memory frontier for complex synapses , 2013, NIPS.

[65]  Chiara Bartolozzi,et al.  Silicon synaptic homeostasis , 2006 .

[66]  Chih-Cheng Chang,et al.  Mitigating Asymmetric Nonlinear Weight Update Effects in Hardware Neural Network Based on Analog Resistive Synapse , 2018, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[67]  Farnood Merrikh-Bayat,et al.  Training and operation of an integrated neuromorphic network based on metal-oxide memristors , 2014, Nature.

[68]  William B Levy,et al.  Energy-Efficient Neuronal Computation via Quantal Synaptic Failures , 2002, The Journal of Neuroscience.

[69]  Jean-Pascal Pfister,et al.  Optimal Spike-Timing-Dependent Plasticity for Precise Action Potential Firing in Supervised Learning , 2005, Neural Computation.

[70]  Wulfram Gerstner,et al.  Neuronal Dynamics: From Single Neurons To Networks And Models Of Cognition , 2014 .

[71]  Philip Heng Wai Leong,et al.  FINN: A Framework for Fast, Scalable Binarized Neural Network Inference , 2016, FPGA.

[72]  Ahmed M. Eltawil,et al.  Independent Component Analysis Using RRAMs , 2019, IEEE Transactions on Nanotechnology.

[73]  Chiara Bartolozzi,et al.  Neuromorphic Electronic Circuits for Building Autonomous Cognitive Systems , 2014, Proceedings of the IEEE.

[74]  Yann LeCun,et al.  Regularization of Neural Networks using DropConnect , 2013, ICML.

[75]  Elisabetta Chicca,et al.  Learning in Silicon Beyond STDP: A Neuromorphic Implementation of Multi-Factor Synaptic Plasticity With Calcium-Based Dynamics , 2016, IEEE Transactions on Circuits and Systems I: Regular Papers.

[76]  Gert Cauwenberghs,et al.  Probabilistic synaptic weighting in a reconfigurable network of VLSI integrate-and-fire neurons , 2001, Neural Networks.

[77]  Jacques Kaiser,et al.  Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE) , 2018, Frontiers in Neuroscience.

[78]  David Kappel,et al.  Network Plasticity as Bayesian Inference , 2015, PLoS Comput. Biol..

[79]  J. Hounsgaard,et al.  Voltage fluctuations in neurons: signal or noise? , 2011, Physiological reviews.

[80]  Peng Lin,et al.  Fully memristive neural networks for pattern classification with unsupervised learning , 2018 .

[81]  Giacomo Indiveri,et al.  A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses , 2015, Front. Neurosci..

[82]  Shimeng Yu,et al.  Neuro-Inspired Computing With Emerging Nonvolatile Memorys , 2018, Proceedings of the IEEE.

[83]  N. Singh,et al.  Physical mechanisms of endurance degradation in TMO-RRAM , 2011, 2011 International Electron Devices Meeting.

[84]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[85]  Jochen Triesch,et al.  Independent Component Analysis in Spiking Neurons , 2010, PLoS Comput. Biol..

[86]  Xiaochen Peng,et al.  Fully parallel RRAM synaptic array for implementing binary neural network with (+1, −1) weights and (+1, 0) neurons , 2018, 2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC).

[87]  Damien Querlioz,et al.  Bioinspired Programming of Memory Devices for Implementing an Inference Engine , 2015, Proceedings of the IEEE.

[88]  P. Pavan,et al.  A Novel Program-Verify Algorithm for Multi-Bit Operation in HfO2 RRAM , 2015, IEEE Electron Device Letters.

[89]  Ahmed M. Eltawil,et al.  Modeling and Analysis of Passive Switching Crossbar Arrays , 2018, IEEE Transactions on Circuits and Systems I: Regular Papers.

[90]  T. Branco,et al.  The probability of neurotransmitter release: variability and feedback control at single synapses , 2009, Nature Reviews Neuroscience.

[91]  Gert Cauwenberghs,et al.  Deep Supervised Learning Using Local Errors , 2017, Front. Neurosci..

[92]  W. Schultz.  Getting Formal with Dopamine and Reward , 2002, Neuron.

[93]  B. Katz.  Nerve, Muscle and Synapse , 1966 .

[94]  Ahmed M. Eltawil,et al.  On Resistive Memories: One Step Row Readout Technique and Sensing Circuitry , 2019, ArXiv.