Partial-Gated Memristor Crossbar for Fast and Power-Efficient Defect-Tolerant Training

A real memristor crossbar has defects, which should be considered during the retraining time after the pre-training of the crossbar. For retraining the crossbar with defects, memristors should be updated with the weights that are calculated by the back-propagation algorithm. Unfortunately, programming the memristors takes a very long time and consumes a large amount of power, because of the incremental behavior of memristor’s program-verify scheme for the fine-tuning of memristor’s conductance. To reduce the programming time and power, the partial gating scheme is proposed here to realize the partial training, where only some part of neurons are trained, which are more responsible in the recognition error. By retraining the part, rather than the entire crossbar, the programming time and power of memristor crossbar can be significantly reduced. The proposed scheme has been verified by CADENCE circuit simulation with the real memristor’s Verilog-A model. When compared to retraining the entire crossbar, the loss of recognition rate of the partial gating scheme has been estimated only as small as 2.5% and 2.9%, for the MNIST and CIFAR-10 datasets, respectively. However, the programming time and power can be saved by 86% and 89.5% than the 100% retraining, respectively.

[1]  Kyeong-Sik Min,et al.  Asymmetrical Training Scheme of Binary-Memristor-Crossbar-Based Neural Networks for Energy-Efficient Edge-Computing Nanoscale Systems , 2019, Micromachines.

[2]  Kyeong-Sik Min,et al.  New pulse amplitude modulation for fine tuning of memristor synapses , 2016, Microelectron. J..

[3]  Fernando Morgado Dias,et al.  Artificial neural networks: a review of commercial hardware , 2004, Eng. Appl. Artif. Intell..

[4]  Young-Ho Lim,et al.  A 3.3 V 32 Mb NAND flash memory with incremental step pulse programming scheme , 1995 .

[5]  Indranil Chakraborty,et al.  Technology Aware Training in Memristive Neuromorphic Systems for Nonideal Synaptic Crossbars , 2017, IEEE Transactions on Emerging Topics in Computational Intelligence.

[6]  Spyros Stathopoulos,et al.  Multibit memory operation of metal-oxide bi-layer memristors , 2017, Scientific Reports.

[7]  Yiran Chen,et al.  Accelerator-friendly neural-network training: Learning variations and defects in RRAM crossbar , 2017, Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017.

[8]  Runze Han,et al.  Design and Hardware Implementation of Neuromorphic Systems With RRAM Synapses and Threshold-Controlled Neurons for Pattern Recognition , 2018, IEEE Transactions on Circuits and Systems I: Regular Papers.

[9]  Gernot A. Fink,et al.  Neuron Pruning for Compressing Deep Networks Using Maxout Architectures , 2017, GCPR.

[10]  Catherine E. Graves,et al.  Memristor‐Based Analog Computation and Neural Network Classification with a Dot Product Engine , 2018, Advanced materials.

[11]  Qing Wu,et al.  Efficient and self-adaptive in-situ learning in multilayer memristor neural networks , 2018, Nature Communications.

[12]  D. Stewart,et al.  The missing memristor found , 2008, Nature.

[13]  Jun Tae Jang,et al.  Effect of oxygen content of the LaAlO 3 layer on the synaptic behavior of Pt/LaAlO 3 /Nb-doped SrTiO 3 memristors for neuromorphic applications , 2017 .

[14]  Pritish Narayanan,et al.  Neuromorphic computing using non-volatile memory , 2017 .

[15]  Jian Cheng,et al.  Quantized Convolutional Neural Networks for Mobile Devices , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Yiran Chen,et al.  A quantization-aware regularized learning method in multilevel memristor-based neuromorphic computing system , 2017, 2017 IEEE 6th Non-Volatile Memory Systems and Applications Symposium (NVMSA).

[17]  Ran El-Yaniv,et al.  Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations , 2016, J. Mach. Learn. Res..

[18]  Kyeong-Sik Min,et al.  New Memristor-Based Crossbar Array Architecture with 50-% Area Reduction and 48-% Power Saving for Matrix-Vector Multiplication of Analog Neuromorphic Computing , 2014 .

[19]  Leon O. Chua,et al.  Neuromemristive Circuits for Edge Computing: A Review , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[20]  Fabrizio Lombardi,et al.  Operational fault detection and monitoring of a memristor-based LUT , 2015, 2015 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[21]  Kyeong-Sik Min,et al.  Memristor Binarized Neural Networks , 2018, JOURNAL OF SEMICONDUCTOR TECHNOLOGY AND SCIENCE.

[22]  Kaushik Roy,et al.  Technology Aware Training in Memristive Neuromorphic Systems based on non-ideal Synaptic Crossbars , 2017, ArXiv.

[23]  Yiran Chen,et al.  Memristor crossbar based hardware realization of BSB recall function , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[24]  Arkady Borisov,et al.  Neurons vs Weights Pruning in Artificial Neural Networks , 2015 .

[25]  K. Yamamori,et al.  Two-stage parallel partial retraining scheme for defective multi-layer neural networks , 2000, Proceedings Fourth International Conference/Exhibition on High Performance Computing in the Asia-Pacific Region.

[26]  Csaba Andras Moritz,et al.  SkyNet: Memristor-based 3D IC for artificial neural networks , 2017, 2017 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH).

[27]  Susumu Horiguchi,et al.  The efficient design of fault-tolerant artificial neural networks , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[28]  Wei Yang Lu,et al.  Nanoscale memristor device as synapse in neuromorphic systems. , 2010, Nano letters.

[29]  Ligang Gao,et al.  High precision tuning of state for memristive devices by adaptable variation-tolerant algorithm , 2011, Nanotechnology.

[30]  Chenchen Liu,et al.  Rescuing memristor-based neuromorphic design with high defects , 2017, 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC).

[31]  Mohammad Alhawari,et al.  MemSens: Memristor-Based Radiation Sensor , 2018, IEEE Sensors Journal.

[32]  Frédéric Pétrot,et al.  Ternary neural networks for resource-efficient AI applications , 2016, 2017 International Joint Conference on Neural Networks (IJCNN).

[33]  J. Yang,et al.  Memristive crossbar arrays for brain-inspired computing , 2019, Nature Materials.

[34]  Sophie Pfeifer.  Vlsi Design Of Non Volatile Memories , 2016 .

[35]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[36]  L. Deng,et al.  The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] , 2012, IEEE Signal Processing Magazine.

[37]  J. Yang,et al.  Three-dimensional crossbar arrays of self-rectifying Si/SiO2/Si memristors , 2017, Nature Communications.

[38]  Jiaming Zhang,et al.  Analogue signal and image processing with large memristor crossbars , 2017, Nature Electronics.

[39]  Farnood Merrikh-Bayat,et al.  3-D Memristor Crossbars for Analog and Neuromorphic Computing Applications , 2017, IEEE Transactions on Electron Devices.

[40]  Stefanos Kaxiras,et al.  Techniques for modulating error resilience in emerging multi-value technologies , 2016, Conf. Computing Frontiers.