Fault-Tolerant Spike Routing Algorithm and Architecture for Three Dimensional NoC-Based Neuromorphic Systems

Neuromorphic computing systems are an emerging field that takes its inspiration from the biological neural architectures and computations inside the mammalian nervous system. The spiking neural networks (SNNs) mimic real biological neural networks by conveying information through the communication of short pulses between neurons. Since each neuron in these networks is connected to thousands of others, high bandwidth is required. Moreover, since the spike times are used to encode information in SNN, very low communication latency is also necessary. On the other hand, the combination of Two-dimensional Networks-on-Chip (2D-NoC) and Three-dimensional Integrated Circuits (3D-ICs) can provide a scalable interconnection fabric in large-scale parallel SNN systems. Although the SNNs have some intrinsic fault-tolerance properties, they are still susceptible to a significant amount of faults; especially, when we talk about integrating the large-scale SNN models in hardware. Consequently, the need for efficient solutions capable of avoiding any malfunctions or inaccuracies, as well as early fault-tolerance assessment, is becoming increasingly necessary for the design of future large-scale reliable neuromorphic systems. This paper first presents an analytical model to assess the effect of faulty connections on the performance of a 3D-NoC-based spiking neural network under different neural network topologies. Second, we present a fault-tolerant shortest-path k-means-based multicast routing algorithm (FTSP-KMCR) and architecture for spike routing in 3D-NoC of spiking neurons (3DFT-SNN). Evaluation results show that the proposed SP-KMCR algorithm reduces the average latency by 12.2% when compared to the previously proposed algorithm. In addition, the proposed fault-tolerant methodology enables the system to sustain correct traffic communication with a fault rate up to 20%, while only suffering 16.23% longer latency and 5.49% extra area cost when compared to the baseline architecture.

[1]  Matthew Cook,et al.  Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[2]  Ben A. Abderazek,et al.  Reliability Assessment and Quantitative Evaluation of Soft-Error Resilient 3D Network-on-Chip Systems , 2016, 2016 IEEE 25th Asian Test Symposium (ATS).

[3]  Bernard Brezzo,et al.  TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip , 2015, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[4]  An-Yeu Wu,et al.  Traffic- and Thermal-Aware Run-Time Thermal Management Scheme for 3D NoC Systems , 2010, 2010 Fourth ACM/IEEE International Symposium on Networks-on-Chip.

[5]  H. Markram.  The Blue Brain Project , 2006, Nature Reviews Neuroscience.

[6]  Ran Ginosar,et al.  Scalable network-on-chip architecture for configurable neural networks , 2011, Microprocess. Microsystems.

[7]  Manfred Glesner,et al.  Adaptive and Deadlock-Free Tree-Based Multicast Routing for Networks-on-Chip , 2010, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[8]  Liam McDaid,et al.  Advancing interconnect density for spiking neural network hardware implementations using traffic-aware adaptive network-on-chip routers , 2012, Neural Networks.

[9]  Hannu Tenhunen,et al.  Path-Based Partitioning Methods for 3D Networks-on-Chip with Minimal Adaptive Routing , 2014, IEEE Transactions on Computers.

[10]  Mehdi Modarressi,et al.  A High-Performance Network-on-Chip Topology for Neuromorphic Architectures , 2017, 22017 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC).

[11]  Bernard Girau,et al.  Fault and Error Tolerance in Neural Networks: A Review , 2017, IEEE Access.

[12]  Robert A. Legenstein,et al.  2007 Special Issue: Edge of chaos and prediction of computational performance for neural circuit models , 2007 .

[13]  Trevor Bekolay,et al.  A Large-Scale Model of the Functioning Brain , 2012, Science.

[14]  A. Hodgkin,et al.  A quantitative description of membrane current and its application to conduction and excitation in nerve , 1990 .

[15]  Raúl Rojas,et al.  Neural Networks - A Systematic Introduction , 1996 .

[16]  Gerard J. M. Smit,et al.  Fixed latency on-chip interconnect for hardware spiking neural network architectures , 2013, Parallel Comput..

[17]  Yiping Dong,et al.  Multiple Network-on-Chip Model for High Performance Neural Network , 2010 .

[18]  Liam McDaid,et al.  Scalable Hierarchical Network-on-Chip Architecture for Spiking Neural Network Hardware Implementations , 2013, IEEE Transactions on Parallel and Distributed Systems.

[19]  Tobi Delbrück,et al.  Orientation-Selective aVLSI Spiking Neurons , 2001, NIPS.

[20]  Michael Pfeiffer,et al.  Deep Learning With Spiking Neurons: Opportunities and Challenges , 2018, Front. Neurosci..

[21]  Abderazek Ben Abdallah.  Advanced Multicore Systems-On-Chip , 2017 .

[22]  David Bol,et al.  A 0.086-mm2 12.7-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28-nm CMOS , 2019, IEEE Trans. Biomed. Circuits Syst..

[23]  Giacomo Indiveri,et al.  A Scalable Multicore Architecture With Heterogeneous Memory Structures for Dynamic Neuromorphic Asynchronous Processors (DYNAPs) , 2017, IEEE Transactions on Biomedical Circuits and Systems.

[24]  Wulfram Gerstner,et al.  SPIKING NEURON MODELS Single Neurons , Populations , Plasticity , 2002 .

[25]  Steve Furber,et al.  Large-scale neuromorphic computing systems , 2016, Journal of neural engineering.

[26]  Dong Xiang,et al.  A New Unicast-Based Multicast Scheme for Network-on-Chip Router and Interconnect Testing , 2016, TODE.

[27]  Masoud Daneshtalab,et al.  Customizing Clos Network-on-Chip for Neural Networks , 2017, IEEE Transactions on Computers.

[28]  Eric Shea-Brown,et al.  Stochastic differential equation models for ion channel noise in Hodgkin-Huxley neurons. , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[29]  Anthony N. Burkitt,et al.  A Review of the Integrate-and-fire Neuron Model: I. Homogeneous Synaptic Input , 2006, Biological Cybernetics.

[30]  Rodrigo Alvarez-Icaza,et al.  Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural Simulations , 2014, Proceedings of the IEEE.

[31]  He Liu,et al.  Optimized Mapping Spiking Neural Networks onto Network-on-Chip , 2016, ICA3PP.

[32]  Eric A. Vittoz,et al.  A communication scheme for analog VLSI perceptive systems , 1995 .

[33]  Andreas G. Andreou,et al.  On fault probabilities and yield models for VLSI neural networks , 1997 .

[34]  Abderazek Ben Abdallah,et al.  Scalable Design Methodology and Online Algorithm for TSV-Cluster Defects Recovery in Highly Reliable 3D-NoC Systems , 2017, IEEE Transactions on Emerging Topics in Computing.

[35]  Johannes Schemmel,et al.  Wafer-scale integration of analog neural networks , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[36]  Abderazek Ben Abdallah,et al.  Analytical performance assessment and high-throughput low-latency spike routing algorithm for spiking neural network systems , 2019, The Journal of Supercomputing.

[37]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[38]  Timothée Masquelier,et al.  Deep Learning in Spiking Neural Networks , 2018, Neural Networks.

[39]  Bin Deng,et al.  Real-Time Neuromorphic System for Large-Scale Conductance-Based Spiking Neural Networks , 2019, IEEE Transactions on Cybernetics.

[40]  Bernabé Linares-Barranco,et al.  Multicasting Mesh AER: A Scalable Assembly Approach for Reconfigurable Neuromorphic Structured AER Systems. Application to ConvNets , 2013, IEEE Transactions on Biomedical Circuits and Systems.

[41]  Ran Ginosar,et al.  Network-on-Chip Architectures for Neural Networks , 2010, 2010 Fourth ACM/IEEE International Symposium on Networks-on-Chip.

[42]  Liam McDaid,et al.  Hardware spiking neural network prototyping and application , 2011, Genetic Programming and Evolvable Machines.

[43]  Steve B. Furber,et al.  A Multicast Routing Scheme for a Universal Spiking Neural Network Architecture , 2010, Comput. J..

[44]  Carver A. Mead,et al.  Neuromorphic electronic systems , 1990, Proc. IEEE.

[45]  Gert Cauwenberghs,et al.  Hierarchical Address Event Routing for Reconfigurable Large-Scale Neuromorphic Systems , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[46]  Steve B. Furber,et al.  The SpiNNaker Project , 2014, Proceedings of the IEEE.

[47]  Kwabena Boahen,et al.  Point-to-point connectivity between neuromorphic chips using address events , 2000 .

[48]  Zhen Zhou,et al.  Modeling and analysis of neuronal membrane electrical activities in 3d neuromorphic computing system , 2017, 2017 IEEE International Symposium on Electromagnetic Compatibility & Signal/Power Integrity (EMCSI).

[49]  Eugene M. Izhikevich,et al.  Simple model of spiking neurons , 2003, IEEE Trans. Neural Networks.

[50]  John Wawrzynek,et al.  A multi-sender asynchronous extension to the AER protocol , 1995, Proceedings Sixteenth Conference on Advanced Research in VLSI.

[51]  Akram Ben Ahmed,et al.  Adaptive fault-tolerant architecture and routing algorithm for reliable many-core 3D-NoC systems , 2016, J. Parallel Distributed Comput..

[52]  Ben A. Abderazek,et al.  Low-Latency K-Means Based Multicast Routing Algorithm and Architecture for Three Dimensional Spiking Neuromorphic Chips , 2019, 2019 IEEE International Conference on Big Data and Smart Computing (BigComp).

[53]  Jim Harkin,et al.  SPANNER: A Self-Repairing Spiking Neural Network Hardware Architecture , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[54]  E. Pasero,et al.  Hw-Sw codesign of a flexible neural controller through a FPGA-based neural network programmed in VHDL , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[55]  Behrooz Parhami.  Exact Formulas for the Average Internode Distance in Mesh and Binary Tree Networks , 2013 .

[56]  Deepak Khosla,et al.  Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition , 2014, International Journal of Computer Vision.

[57]  Matthew Cook,et al.  Unsupervised learning of digit recognition using spike-timing-dependent plasticity , 2015, Front. Comput. Neurosci..

[58]  J. Milton.  Dynamics of small neural populations , 1996 .

[59]  Olivier Temam,et al.  The improbable but highly appropriate marriage of 3D stacking and neuromorphic accelerators , 2014, 2014 International Conference on Compilers, Architecture and Synthesis for Embedded Systems (CASES).

[60]  Liam McDaid,et al.  On the role of astroglial syncytia in self-repairing spiking neural networks , 2015, IEEE Transactions on Neural Networks and Learning Systems.