Analytical performance assessment and high-throughput low-latency spike routing algorithm for spiking neural network systems

Large-scale artificial neural networks (ANNs) have been used to mimic the information processing function of the brain. Spiking neural networks (SNNs) are a kind of ANN, which mimic real biological neural networks, conveying information through the communication of short pulses between neurons. Since each neuron in these networks is connected to thousands of others, high bandwidth is required. Moreover, since the spike times are used to encode information in SNN, very low communication latency is also required. The 2D-NoC was used as a solution to provide a scalable interconnection fabric in large-scale parallel SNN systems. The 3D-ICs have also attracted a lot of attention as a potential solution to resolve the interconnect bottleneck. The combination of these two emerging technologies provides a new horizon for IC designs to satisfy the high requirements of low-power and small footprint in emerging AI applications. This paper first presents an analytical model to analyze the performance of different neural network topologies and compare it with a system-level simulation. Second, we present an architecture and a low-latency routing algorithm for spike traffic routing in 3D-NoC of spiking neurons (3DNoC-SNN). The 3DNoC-SNN is validated based on an RTL-level implementation, while area/power analysis is performed using 45-nm CMOS technology.

[1]  Masoud Daneshtalab,et al.  Customizing Clos Network-on-Chip for Neural Networks , 2017, IEEE Transactions on Computers.

[2]  Steve B. Furber,et al.  Neural Systems Engineering , 2008, Computational Intelligence: A Compendium.

[3]  Ben A. Abderazek,et al.  Efficient Optimization and Hardware Acceleration of CNNs towards the Design of a Scalable Neuro inspired Architecture in Hardware , 2018, 2018 IEEE International Conference on Big Data and Smart Computing (BigComp).

[4]  J. Gore,et al.  The Impact of Alzheimer's Disease on the Resting State Functional Connectivity of Brain Regions Modulating Pain: A Cross Sectional Study. , 2017, Journal of Alzheimer's disease : JAD.

[5]  Deepak Khosla,et al.  Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition , 2014, International Journal of Computer Vision.

[6]  Ran Ginosar,et al.  Scalable network-on-chip architecture for configurable neural networks , 2011, Microprocess. Microsystems.

[7]  Steve Furber,et al.  Large-scale neuromorphic computing systems , 2016, Journal of neural engineering.

[8]  Dong Xiang,et al.  A New Unicast-Based Multicast Scheme for Network-on-Chip Router and Interconnect Testing , 2016, TODE.

[9]  Raúl Rojas,et al.  Neural Networks - A Systematic Introduction , 1996 .

[10]  H. Markram.  The Blue Brain Project , 2006, Nature Reviews Neuroscience.

[11]  Eric A. Vittoz,et al.  A communication scheme for analog VLSI perceptive systems , 1995 .

[12]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[13]  Liam McDaid,et al.  Hierarchical Network-on-Chip and Traffic Compression for Spiking Neural Network Implementations , 2012, 2012 IEEE/ACM Sixth International Symposium on Networks-on-Chip.

[14]  Behrooz Parhami.  Exact Formulas for the Average Internode Distance in Mesh and Binary Tree Networks , 2013 .

[15]  Abderazek Ben Abdallah.  Advanced Multicore Systems-On-Chip , 2017 .

[16]  Rodrigo Alvarez-Icaza,et al.  Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural Simulations , 2014, Proceedings of the IEEE.

[17]  Yiping Dong,et al.  Multiple Network-on-Chip Model for High Performance Neural Network , 2010 .

[18]  Liam McDaid,et al.  Scalable Hierarchical Network-on-Chip Architecture for Spiking Neural Network Hardware Implementations , 2013, IEEE Transactions on Parallel and Distributed Systems.

[19]  A. Kauffmann-Zeh.  Nerve repair: Ins and outs of Nogo , 2001, Nature Reviews Neuroscience.

[20]  Wofgang Maas,et al.  Networks of spiking neurons: the third generation of neural network models , 1997 .

[21]  Xiaola Lin,et al.  Multicast Communication in Multicomputer Networks , 1993, ICPP.

[22]  Bin Deng,et al.  Real-Time Neuromorphic System for Large-Scale Conductance-Based Spiking Neural Networks , 2019, IEEE Transactions on Cybernetics.

[23]  Eric Shea-Brown,et al.  Stochastic differential equation models for ion channel noise in Hodgkin-Huxley neurons. , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[24]  Johannes Schemmel,et al.  Wafer-scale integration of analog neural networks , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[25]  Steve B. Furber,et al.  The SpiNNaker Project , 2014, Proceedings of the IEEE.

[26]  Manfred Glesner,et al.  Adaptive and Deadlock-Free Tree-Based Multicast Routing for Networks-on-Chip , 2010, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[27]  Liam McDaid,et al.  Advancing interconnect density for spiking neural network hardware implementations using traffic-aware adaptive network-on-chip routers , 2012, Neural Networks.

[28]  Hannu Tenhunen,et al.  Path-Based Partitioning Methods for 3D Networks-on-Chip with Minimal Adaptive Routing , 2014, IEEE Transactions on Computers.

[29]  Olivier Temam,et al.  The improbable but highly appropriate marriage of 3D stacking and neuromorphic accelerators , 2014, 2014 International Conference on Compilers, Architecture and Synthesis for Embedded Systems (CASES).

[30]  Mancia Anguita,et al.  A Spiking Neural Simulator Integrating Event-Driven and Time-Driven Computation Schemes Using Parallel CPU-GPU Co-Processing: A Case Study , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[31]  E. Pasero,et al.  Hw-Sw codesign of a flexible neural controller through a FPGA-based neural network programmed in VHDL , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[32]  Tobi Delbrück,et al.  Orientation-Selective aVLSI Spiking Neurons , 2001, NIPS.

[33]  Liam McDaid,et al.  Scalable Networks-on-Chip Interconnected Architecture for Astrocyte-Neuron Networks , 2016, IEEE Transactions on Circuits and Systems I: Regular Papers.

[34]  Liam McDaid,et al.  EMBRACE-SysC for analysis of NoC-based Spiking Neural Network architectures , 2010, 2010 International Symposium on System on Chip.

[35]  Zhen Zhou,et al.  Modeling and analysis of neuronal membrane electrical activities in 3d neuromorphic computing system , 2017, 2017 IEEE International Symposium on Electromagnetic Compatibility & Signal/Power Integrity (EMCSI).

[36]  John Wawrzynek,et al.  A multi-sender asynchronous extension to the AER protocol , 1995, Proceedings Sixteenth Conference on Advanced Research in VLSI.

[37]  Akram Ben Ahmed,et al.  Adaptive fault-tolerant architecture and routing algorithm for reliable many-core 3D-NoC systems , 2016, J. Parallel Distributed Comput..

[38]  Liam McDaid,et al.  Hardware spiking neural network prototyping and application , 2011, Genetic Programming and Evolvable Machines.

[39]  Ben A. Abderazek,et al.  Reliability Assessment and Quantitative Evaluation of Soft-Error Resilient 3D Network-on-Chip Systems , 2016, 2016 IEEE 25th Asian Test Symposium (ATS).

[40]  S. Strogatz.  Exploring complex networks , 2001, Nature.

[41]  Kwabena Boahen,et al.  Point-to-point connectivity between neuromorphic chips using address events , 2000 .

[42]  Bernard Brezzo,et al.  TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip , 2015, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[43]  Ran Ginosar,et al.  Network-on-Chip Architectures for Neural Networks , 2010, 2010 Fourth ACM/IEEE International Symposium on Networks-on-Chip.

[44]  Mehdi Modarressi,et al.  A High-Performance Network-on-Chip Topology for Neuromorphic Architectures , 2017, 22017 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC).

[45]  Robert A. Legenstein,et al.  2007 Special Issue: Edge of chaos and prediction of computational performance for neural circuit models , 2007 .

[46]  Abderazek Ben Abdallah,et al.  Scalable Design Methodology and Online Algorithm for TSV-Cluster Defects Recovery in Highly Reliable 3D-NoC Systems , 2017, IEEE Transactions on Emerging Topics in Computing.