Neuro-inspired computing chips

The rapid development of artificial intelligence (AI) demands the rapid development of domain-specific hardware specifically designed for AI applications. Neuro-inspired computing chips integrate a range of features inspired by neurobiological systems and could provide an energy-efficient approach to AI computing workloads. Here, we review the development of neuro-inspired computing chips, including artificial neural network chips and spiking neural network chips. We propose four key metrics for benchmarking neuro-inspired computing chips — computing density, energy efficiency, computing accuracy, and on-chip learning capability — and discuss co-design principles, from the device to the algorithm level, for neuro-inspired computing chips based on non-volatile memory. We also provide a future electronic design automation tool chain and propose a roadmap for the development of large-scale neuro-inspired computing chips. This Review Article examines the development of neuro-inspired computing chips and their key benchmarking metrics, providing a co-design tool chain and proposing a roadmap for future large-scale chips.

[1]  B. Widrow,et al.  Birth, Life, and Death in Microelectronic Systems , 1961, IRE Transactions on Military Electronics.

[2]  Lawrence D. Jackel,et al.  Artificial neural networks for computing , 1986 .

[3]  Carver A. Mead,et al.  Neuromorphic electronic systems , 1990, Proc. IEEE.

[4]  L. Abbott,et al.  Competitive Hebbian learning through spike-timing-dependent synaptic plasticity , 2000, Nature Neuroscience.

[5]  D. Stewart,et al.  The missing memristor found , 2008, Nature.

[6]  Wei Yang Lu,et al.  Nanoscale memristor device as synapse in neuromorphic systems. , 2010, Nano letters.

[7]  Johannes Schemmel,et al.  A wafer-scale neuromorphic hardware system for large-scale neural modeling , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[8]  Yong Liu,et al.  A 45nm CMOS neuromorphic chip with a scalable architecture for learning in networks of spiking neurons , 2011, 2011 IEEE Custom Integrated Circuits Conference (CICC).

[9]  Farnood Merrikh-Bayat,et al.  Memristive Neuro-Fuzzy System , 2013, IEEE Transactions on Cybernetics.

[10]  Eitan Yaakobi,et al.  Sneak-path constraints in memristor crossbar arrays , 2013, 2013 IEEE International Symposium on Information Theory.

[11]  Rodrigo Alvarez-Icaza,et al.  Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural Simulations , 2014, Proceedings of the IEEE.

[12]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[13]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[14]  Giacomo Indiveri,et al.  A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses , 2015, Front. Neurosci..

[15]  Zhengya Zhang,et al.  A 640M pixel/s 3.65mW sparse event-driven neuromorphic object recognition processor with on-chip learning , 2015, 2015 Symposium on VLSI Circuits (VLSI Circuits).

[16]  Dmitri Strukov,et al.  Manhattan rule training for memristive crossbar circuit pattern classifiers , 2015, 2015 IEEE 9th International Symposium on Intelligent Signal Processing (WISP) Proceedings.

[17]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[18]  Farnood Merrikh-Bayat,et al.  Training and operation of an integrated neuromorphic network based on metal-oxide memristors , 2014, Nature.

[19]  R. Jordan,et al.  NVM neuromorphic core with 64k-cell (256-by-256) phase change memory synaptic array with on-chip neuron circuits for continuous in-situ learning , 2015, 2015 IEEE International Electron Devices Meeting (IEDM).

[20]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[21]  Zhiwei Li,et al.  Binary neural network with 16 Mb RRAM macro chip for classification and online training , 2016, 2016 IEEE International Electron Devices Meeting (IEDM).

[22]  Vivienne Sze,et al.  14.5 Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks , 2016, ISSCC.

[23]  Andrew S. Cassidy,et al.  Convolutional networks for fast, energy-efficient neuromorphic computing , 2016, Proceedings of the National Academy of Sciences.

[24]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Catherine Graves,et al.  Dot-product engine for neuromorphic computing: Programming 1T1M crossbar to accelerate matrix-vector multiplication , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[26]  Ru Huang,et al.  Engineering incremental resistive switching in TaOx based memristors for brain-inspired computing. , 2016, Nanoscale.

[27]  Yu Wang,et al.  PRIME: A Novel Processing-in-Memory Architecture for Neural Network Computation in ReRAM-Based Main Memory , 2016, 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA).

[28]  Jun-Seok Park,et al.  14.6 A 1.42TOPS/W deep convolutional neural network recognition processor for intelligent IoE systems , 2016, 2016 IEEE International Solid-State Circuits Conference (ISSCC).

[29]  Miao Hu,et al.  ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars , 2016, 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA).

[30]  Roger Melko,et al.  Quantum Boltzmann Machine , 2016, 1601.02036.

[31]  Ali Khiat,et al.  Unsupervised learning in probabilistic neural networks with multi-state metal-oxide memristive synapses , 2016, Nature Communications.

[32]  Shimeng Yu,et al.  Ferroelectric FET analog synapse for acceleration of deep neural network training , 2017, 2017 IEEE International Electron Devices Meeting (IEDM).

[33]  Yuan Xie,et al.  DRISA: A DRAM-based Reconfigurable In-Situ Accelerator , 2017, 2017 50th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

[34]  Yiran Chen,et al.  PipeLayer: A Pipelined ReRAM-Based Accelerator for Deep Learning , 2017, 2017 IEEE International Symposium on High Performance Computer Architecture (HPCA).

[35]  Shimeng Yu,et al.  NbOx based oscillation neuron for neuromorphic computing , 2017 .

[36]  F. Merrikh Bayat,et al.  Fast, energy-efficient, robust, and reproducible mixed-signal neuromorphic classifier based on embedded NOR flash memory technology , 2017, 2017 IEEE International Electron Devices Meeting (IEDM).

[37]  Robert A. Legenstein,et al.  Neuromorphic hardware in the loop: Training a deep spiking network on the BrainScaleS wafer-scale system , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[38]  Wei D. Lu,et al.  Sparse coding with memristor networks. , 2017, Nature nanotechnology.

[39]  V. Sze,et al.  Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks , 2016, IEEE Journal of Solid-State Circuits.

[40]  J. Yang,et al.  Memristors with diffusive dynamics as synaptic emulators for neuromorphic computing. , 2017, Nature materials.

[41]  H.-S. Philip Wong,et al.  Device and circuit optimization of RRAM for neuromorphic computing , 2017, 2017 IEEE International Electron Devices Meeting (IEDM).

[42]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[43]  Shimeng Yu,et al.  Investigation of statistical retention of filamentary analog RRAM for neuromophic computing , 2017, 2017 IEEE International Electron Devices Meeting (IEDM).

[44]  Zhuo Wang,et al.  In-Memory Computation of a Machine-Learning Classifier in a Standard 6T SRAM Array , 2017, IEEE Journal of Solid-State Circuits.

[45]  H.-S. Philip Wong,et al.  Face classification using electronic synapses , 2017, Nature Communications.

[46]  F. Merrikh Bayat,et al.  Spike-timing-dependent plasticity learning of coincidence detection with passively integrated memristive circuits , 2018, Nature Communications.

[47]  Mohammed A. Zidan,et al.  Parasitic Effect Analysis in Memristor-Array-Based Neuromorphic Systems , 2018, IEEE Transactions on Nanotechnology.

[48]  Catherine E. Graves,et al.  Memristor‐Based Analog Computation and Neural Network Classification with a Dot Product Engine , 2018, Advanced materials.

[49]  Douglas M. Bishop,et al.  ECRAM as Scalable Synaptic Cell for High-Speed, Low-Power Neuromorphic Computing , 2018, 2018 IEEE International Electron Devices Meeting (IEDM).

[50]  Weiwei Xia,et al.  Memristor Crossbars with 4.5 Terabits-per-Inch-Square Density and Two Nanometer Dimension , 2018, ArXiv.

[51]  Jason Cong,et al.  Scaling for edge inference of deep neural networks , 2018 .

[52]  Alessandro Calderoni,et al.  Learning of spatiotemporal patterns in a spiking neural network with resistive switching synapses , 2018, Science Advances.

[53]  Peng Lin,et al.  Fully memristive neural networks for pattern classification with unsupervised learning , 2018 .

[54]  Sujan Kumar Gonugondla,et al.  A Multi-Functional In-Memory Inference Processor Using a Standard 6T SRAM Array , 2018, IEEE Journal of Solid-State Circuits.

[55]  Qing Wu,et al.  Efficient and self-adaptive in-situ learning in multilayer memristor neural networks , 2018, Nature Communications.

[56]  Pritish Narayanan,et al.  Equivalent-accuracy accelerated neural-network training using analogue memory , 2018, Nature.

[57]  Ryutaro Yasuhara,et al.  A 4M Synapses integrated Analog ReRAM based 66.5 TOPS/W Neural-Network Processor with Cell Current Controlled Writing and Flexible Network Architecture , 2018, 2018 IEEE Symposium on VLSI Technology.

[58]  Xiaoyu Sun,et al.  Characterizing Endurance Degradation of Incremental Switching in Analog RRAM for Neuromorphic Systems , 2018, 2018 IEEE International Electron Devices Meeting (IEDM).

[59]  Bo Chen,et al.  Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[60]  Yusuf Leblebici,et al.  Neuromorphic computing with multi-memristive synapses , 2017, Nature Communications.

[61]  Shimeng Yu,et al.  Neuro-Inspired Computing With Emerging Nonvolatile Memorys , 2018, Proceedings of the IEEE.

[62]  Sujan Kumar Gonugondla,et al.  A 42pJ/decision 3.12TOPS/W robust in-memory machine learning classifier with on-chip training , 2018, 2018 IEEE International Solid - State Circuits Conference - (ISSCC).

[63]  Alex Pappachen James,et al.  Hierarchical Temporal Memory Using Memristor Networks: A Survey , 2018, IEEE Transactions on Emerging Topics in Computational Intelligence.

[64]  Shimeng Yu,et al.  A ferroelectric field effect transistor based synaptic weight cell , 2018, Journal of Physics D: Applied Physics.

[65]  Meng-Fan Chang,et al.  A 65nm 4Kb algorithm-dependent computing-in-memory SRAM unit-macro with 2.3ns and 55.8TOPS/W fully parallel product-sum operation for binary DNN edge processors , 2018, 2018 IEEE International Solid - State Circuits Conference - (ISSCC).

[66]  Anantha Chandrakasan,et al.  Conv-RAM: An energy-efficient SRAM with embedded convolution computation for low-power CNN-based machine learning applications , 2018, 2018 IEEE International Solid - State Circuits Conference - (ISSCC).

[67]  Xiaochen Peng,et al.  XNOR-RRAM: A scalable and parallel resistive synaptic architecture for binary neural networks , 2018, 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[68]  Wenqiang Zhang,et al.  Sign backpropagation: An on-chip learning algorithm for analog RRAM neuromorphic computing systems , 2018, Neural Networks.

[69]  Hong Wang,et al.  Loihi: A Neuromorphic Manycore Processor with On-Chip Learning , 2018, IEEE Micro.

[70]  Shimeng Yu,et al.  Exploiting Hybrid Precision for Training and Inference: A 2T-1FeFET Based Analog Synaptic Weight Cell , 2018, 2018 IEEE International Electron Devices Meeting (IEDM).

[71]  Giacomo Indiveri,et al.  A Scalable Multicore Architecture With Heterogeneous Memory Structures for Dynamic Neuromorphic Asynchronous Processors (DYNAPs) , 2017, IEEE Transactions on Biomedical Circuits and Systems.

[72]  Meng-Fan Chang,et al.  A 65nm 1Mb nonvolatile computing-in-memory ReRAM macro with sub-16ns multiply-and-accumulate for binary DNN AI edge processors , 2018, 2018 IEEE International Solid - State Circuits Conference - (ISSCC).

[73]  Michael L. Schneider,et al.  Ultralow power artificial synapses using nanotextured magnetic Josephson junctions , 2018, Science Advances.

[74]  J. Yang,et al.  Memristor crossbar arrays with 6-nm half-pitch and 2-nm critical dimension , 2018, Nature Nanotechnology.

[75]  Xiaochen Peng,et al.  NeuroSim: A Circuit-Level Macro Model for Benchmarking Neuro-Inspired Architectures in Online Learning , 2018, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[76]  Jiaming Zhang,et al.  Analogue signal and image processing with large memristor crossbars , 2017, Nature Electronics.

[77]  Bin Gao,et al.  Multiplication on the edge , 2018 .

[78]  John C. Platt,et al.  Quantum supremacy using a programmable superconducting processor , 2019, Nature.

[79]  Kaushik Roy,et al.  Towards spike-based machine intelligence with neuromorphic computing , 2019, Nature.

[80]  Jun Yang,et al.  24.4 Sandwich-RAM: An Energy-Efficient In-Memory BWN Architecture with Pulse-Width Modulation , 2019, 2019 IEEE International Solid- State Circuits Conference - (ISSCC).

[81]  David Bol,et al.  A 0.086-mm$^2$ 12.7-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28-nm CMOS , 2018, IEEE Transactions on Biomedical Circuits and Systems.

[82]  Xiaochen Peng,et al.  Design Guidelines of RRAM based Neural-Processing-Unit: A Joint Device-Circuit-Algorithm Analysis , 2019, 2019 56th ACM/IEEE Design Automation Conference (DAC).

[83]  Yuan Xie,et al.  FPSA: A Full System Stack Solution for Reconfigurable ReRAM-based NN Accelerator Architecture , 2019, ASPLOS.

[84]  Peng Lin,et al.  Reinforcement learning with analogue memristor arrays , 2019, Nature Electronics.

[85]  Qing Wu,et al.  Long short-term memory networks in memristor crossbar arrays , 2018, Nature Machine Intelligence.

[86]  Mingguo Zhao,et al.  Towards artificial general intelligence with hybrid Tianjic chip architecture , 2019, Nature.

[87]  Mary Wootters,et al.  The N3XT Approach to Energy-Efficient Abundant-Data Computing , 2019, Proceedings of the IEEE.

[88]  Zhengya Zhang,et al.  A fully integrated reprogrammable memristor–CMOS system for efficient multiply–accumulate operations , 2019, Nature Electronics.

[89]  Daniele Ielmini,et al.  Solving matrix equations in one step with cross-point resistive arrays , 2019, Proceedings of the National Academy of Sciences.

[90]  David Bol,et al.  MorphIC: A 65-nm 738k-Synapse/mm$^2$ Quad-Core Binary-Weight Digital Neuromorphic Processor With Stochastic Spike-Driven Online Learning , 2019, IEEE Transactions on Biomedical Circuits and Systems.

[91]  M. R. Mahmoodi,et al.  Versatile stochastic dot product circuits based on nonvolatile memories for high performance neurocomputing and neurooptimization , 2019, Nature Communications.

[92]  Chia-Lin Yang,et al.  Sparse ReRAM Engine: Joint Exploration of Activation and Weight Sparsity in Compressed Neural Networks , 2019, 2019 ACM/IEEE 46th Annual International Symposium on Computer Architecture (ISCA).

[93]  Song Han,et al.  A Configurable Multi-Precision CNN Computing Framework Based on Single Bit RRAM , 2019, 2019 56th ACM/IEEE Design Automation Conference (DAC).

[94]  Dongsuk Jeon,et al.  7.6 A 65nm 236.5nJ/Classification Neuromorphic Processor with 7.5% Energy Overhead On-Chip Learning Using Direct Spike-Only Feedback , 2019, 2019 IEEE International Solid- State Circuits Conference - (ISSCC).

[95]  Dejan S. Milojicic,et al.  PUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inference , 2019, ASPLOS.

[96]  Meng-Fan Chang,et al.  Monolithic 3D+ -IC based Reconfigurable Compute-in-Memory SRAM Macro , 2019, 2019 Symposium on VLSI Technology.

[97]  Hoi-Jun Yoo,et al.  1.2 Intelligence on Silicon: From Deep-Neural-Network Accelerators to Brain Mimicking AI-SoCs , 2019, 2019 IEEE International Solid- State Circuits Conference - (ISSCC).

[98]  Meng-Fan Chang,et al.  24.5 A Twin-8T SRAM Computation-In-Memory Macro for Multiple-Bit CNN-Based Machine Learning , 2019, 2019 IEEE International Solid- State Circuits Conference - (ISSCC).

[99]  Meng-Fan Chang,et al.  24.1 A 1Mb Multibit ReRAM Computing-In-Memory Macro with 14.6ns Parallel MAC Computing Time for CNN Based AI Edge Processors , 2019, 2019 IEEE International Solid- State Circuits Conference - (ISSCC).

[100]  Meng-Fan Chang,et al.  33.2 A Fully Integrated Analog ReRAM Based 78.4TOPS/W Compute-In-Memory Chip with Fully Parallel MAC Computing , 2020, 2020 IEEE International Solid- State Circuits Conference - (ISSCC).

[101]  E. Eleftheriou,et al.  Memory devices and applications for in-memory computing , 2020, Nature Nanotechnology.

[102]  Mark Barnell,et al.  Three-dimensional memristor circuits as complex neural networks , 2020, Nature Electronics.

[103]  Bin Gao,et al.  Fully hardware-implemented memristor convolutional neural network , 2020, Nature.

[104]  Wang,et al.  In-Datacenter Performance Analysis of a Tensor Processing UnitTM , .