Asymmetrical Training Scheme of Binary-Memristor-Crossbar-Based Neural Networks for Energy-Efficient Edge-Computing Nanoscale Systems

For realizing neural networks with binary memristor crossbars, memristors should be programmed by high-resistance state (HRS) and low-resistance state (LRS), according to the training algorithms like backpropagation. Unfortunately, it takes a very long time and consumes a large amount of power in training the memristor crossbar, because the program-verify scheme of memristor-programming is based on the incremental programming pulses, where many programming and verifying pulses are repeated until the target conductance. Thus, this reduces the programming time and power is very essential for energy-efficient and fast training of memristor networks. In this paper, we compared four different programming schemes, which are F-F, C-F, F-C, and C-C, respectively. C-C means both HRS and LRS are coarse-programmed. C-F has the coarse-programmed HRS and fine LRS, respectively. F-C is vice versa of C-F. In F-F, both HRS and LRS are fine-programmed. Comparing the error-energy products among the four schemes, C-F shows the minimum error with the minimum energy consumption. The asymmetrical coarse HRS and fine LRS can reduce the time and energy during the crossbar training significantly, because only LRS is fine-programmed. Moreover, the asymmetrical C-F can maintain the network’s error as small as F-F, which is due to the coarse-programmed HRS that slightly degrades the error.

[1]  Dexian Huang,et al.  Data-driven soft sensor development based on deep learning technique , 2014 .

[2]  Seok-Jin Ham,et al.  Neuromorphic crossbar circuit with nanoscale filamentary-switching binary memristors for speech recognition , 2014, Nanoscale Research Letters.

[3]  Ran El-Yaniv,et al.  Binarized Neural Networks , 2016, NIPS.

[4]  Csaba Andras Moritz,et al.  SkyNet: Memristor-based 3D IC for artificial neural networks , 2017, 2017 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH).

[5]  Eriko Nurvitadhi,et al.  Accelerating Binarized Neural Networks: Comparison of FPGA, CPU, GPU, and ASIC , 2016, 2016 International Conference on Field-Programmable Technology (FPT).

[6]  Mohammad Alhawari,et al.  MemSens: Memristor-Based Radiation Sensor , 2018, IEEE Sensors Journal.

[7]  Onur Mutlu,et al.  Threshold voltage distribution in MLC NAND flash memory: Characterization, analysis, and modeling , 2013, 2013 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[8]  Jun Tae Jang,et al.  Effect of oxygen content of the LaAlO 3 layer on the synaptic behavior of Pt/LaAlO 3 /Nb-doped SrTiO 3 memristors for neuromorphic applications , 2017 .

[9]  Indranil Chakraborty,et al.  Technology Aware Training in Memristive Neuromorphic Systems for Nonideal Synaptic Crossbars , 2017, IEEE Transactions on Emerging Topics in Computational Intelligence.

[10]  Schahram Dustdar,et al.  Going Back to the Roots—The Evolution of Edge Computing, An IoT Perspective , 2018, IEEE Internet Computing.

[11]  L. Deng,et al.  The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] , 2012, IEEE Signal Processing Magazine.

[12]  J. Yang,et al.  Three-dimensional crossbar arrays of self-rectifying Si/SiO2/Si memristors , 2017, Nature Communications.

[13]  Jiaming Zhang,et al.  Analogue signal and image processing with large memristor crossbars , 2017, Nature Electronics.

[14]  Nirwan Ansari,et al.  EdgeIoT: Mobile Edge Computing for the Internet of Things , 2016, IEEE Communications Magazine.

[15]  Farnood Merrikh-Bayat,et al.  3-D Memristor Crossbars for Analog and Neuromorphic Computing Applications , 2017, IEEE Transactions on Electron Devices.

[16]  Qi Liu,et al.  Multilevel resistive switching with ionic and metallic filaments , 2009 .

[17]  Nicholas Soures,et al.  Robustness of a memristor based liquid state machine , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[18]  Young-Ho Lim,et al.  A 3.3 V 32 Mb NAND flash memory with incremental step pulse programming scheme , 1995 .

[19]  Leon O. Chua,et al.  Neuromemristive Circuits for Edge Computing: A Review , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[20]  Kyeong-Sik Min,et al.  New pulse amplitude modulation for fine tuning of memristor synapses , 2016, Microelectron. J..

[21]  Chris Yakopcic,et al.  On-chip training of memristor crossbar based multi-layer neural networks , 2017, Microelectron. J..

[22]  Yu Wang,et al.  The stochastic modeling of TiO2 memristor and its usage in neuromorphic system design , 2014, 2014 19th Asia and South Pacific Design Automation Conference (ASP-DAC).

[23]  Michael F. Shlesinger,et al.  Time‐Scale Invariance in Transport and Relaxation , 1991 .

[24]  Yu Wang,et al.  Switched by input: Power efficient structure for RRAM-based convolutional neural network , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[25]  Sungjun Kim,et al.  Beyond SiOx: an active electronics resurgence and biomimetic reactive oxygen species production and regulation from mitochondria , 2018 .

[26]  Kyeong-Sik Min,et al.  Two-Step Write Scheme for Reducing Sneak-Path Leakage in Complementary Memristor Array , 2012, IEEE Transactions on Nanotechnology.

[27]  Kyeong-Sik Min,et al.  In-DRAM bitwise processing circuit for low-power and fast computation , 2017 .

[28]  Yiran Chen,et al.  The applications of NVM technology in hardware security , 2016, 2016 International Great Lakes Symposium on VLSI (GLSVLSI).

[29]  Emmanuelle J. Merced-Grafals,et al.  Repeatable, accurate, and high speed multi-level programming of memristor 1T1R arrays for power efficient analog computing applications , 2016, Nanotechnology.

[30]  Wei Lu,et al.  Short-term Memory to Long-term Memory Transition in a Nanoscale Memristor , 2022 .

[31]  Wei Zhang,et al.  Nonvolatile CBRAM-Crossbar-Based 3-D-Integrated Hybrid Memory for Data Retention , 2014, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[32]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[33]  Derek Abbott,et al.  Memristive crypto primitive for building highly secure physical unclonable functions , 2015, Scientific Reports.

[34]  J. C. Phillips,et al.  Stretched exponential relaxation in molecular and electronic glasses , 1996 .

[35]  Geoffrey Zweig,et al.  Recent advances in deep learning for speech research at Microsoft , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[36]  Tarik Taleb,et al.  Edge Computing for the Internet of Things: A Case Study , 2018, IEEE Internet of Things Journal.

[37]  Ran El-Yaniv,et al.  Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations , 2016, J. Mach. Learn. Res..

[38]  Sung-Bae Cho,et al.  Human activity recognition with smartphone sensors using deep learning neural networks , 2016, Expert Syst. Appl..

[39]  Byung-Gook Park,et al.  Analog Synaptic Behavior of a Silicon Nitride Memristor. , 2017, ACS applied materials & interfaces.

[40]  Kyeong-Sik Min,et al.  Memristor Binarized Neural Networks , 2018, JOURNAL OF SEMICONDUCTOR TECHNOLOGY AND SCIENCE.

[41]  Tao Zhang,et al.  Overcoming the challenges of crossbar resistive memory architectures , 2015, 2015 IEEE 21st International Symposium on High Performance Computer Architecture (HPCA).

[42]  Rajiv V. Joshi,et al.  An energy-efficient matrix multiplication accelerator by distributed in-memory computing on binary RRAM crossbar , 2016, 2016 21st Asia and South Pacific Design Automation Conference (ASP-DAC).

[43]  Farnood Merrikh-Bayat,et al.  Training and operation of an integrated neuromorphic network based on metal-oxide memristors , 2014, Nature.

[44]  Wei Yang Lu,et al.  Nanoscale memristor device as synapse in neuromorphic systems. , 2010, Nano letters.

[45]  Yi Li,et al.  Neuronal dynamics in HfOx/AlOy-based homeothermic synaptic memristors with low-power and homogeneous resistive switching. , 2018, Nanoscale.

[46]  Catherine Graves,et al.  Dot-product engine for neuromorphic computing: Programming 1T1M crossbar to accelerate matrix-vector multiplication , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[47]  Ligang Gao,et al.  Demonstration of Convolution Kernel Operation on Resistive Cross-Point Array , 2016, IEEE Electron Device Letters.

[48]  D. Stewart,et al.  The missing memristor found , 2008, Nature.

[49]  Kaushik Roy,et al.  Technology Aware Training in Memristive Neuromorphic Systems based on non-ideal Synaptic Crossbars , 2017, ArXiv.