Hardware implementation of RRAM based binarized neural networks

Resistive switching random access memory (RRAM) has been explored to accelerate the computation of neural networks. RRAM with linear conductance modulation is usually required for the efficient weight updating during the online training according to the back-propagation algorithm. However, most RRAM devices usually show the nonlinear characteristic. Here, to overcome the dilemma, we designed a novel weight updating principle for binarized neural networks, which enables the nonlinear RRAM to realize the weight updating in efficiency during online training. Moreover, a vector-matrix multiplication is designed to parallel calculate the dot-products of the forward and backward propagation. 1 kb nonlinear RRAM array is fabricated to demonstrate the feasibility of the analog accumulation and the parallel vector-matrix multiplication. The results achieved in this work offer new solutions for future energy efficient neural networks.Resistive switching random access memory (RRAM) has been explored to accelerate the computation of neural networks. RRAM with linear conductance modulation is usually required for the efficient weight updating during the online training according to the back-propagation algorithm. However, most RRAM devices usually show the nonlinear characteristic. Here, to overcome the dilemma, we designed a novel weight updating principle for binarized neural networks, which enables the nonlinear RRAM to realize the weight updating in efficiency during online training. Moreover, a vector-matrix multiplication is designed to parallel calculate the dot-products of the forward and backward propagation. 1 kb nonlinear RRAM array is fabricated to demonstrate the feasibility of the analog accumulation and the parallel vector-matrix multiplication. The results achieved in this work offer new solutions for future energy efficient neural networks.

[1]  D. Querlioz,et al.  Visual Pattern Extraction Using Energy-Efficient “2-PCM Synapse” Neuromorphic Architecture , 2012, IEEE Transactions on Electron Devices.

[2]  Yu-Fen Wang,et al.  Characterization and Modeling of Nonfilamentary Ta/TaOx/TiO2/Ti Analog Synaptic Device , 2015, Scientific Reports.

[3]  Qi Liu,et al.  Super non-linear RRAM with ultra-low power for 3D vertical nano-crossbar arrays. , 2016, Nanoscale.

[4]  Ligang Gao,et al.  Programming Protocol Optimization for Analog Weight Tuning in Resistive Memories , 2015, IEEE Electron Device Letters.

[5]  Peng Huang,et al.  Compact Model of HfOX-Based Electronic Synaptic Devices for Neuromorphic Computing , 2017, IEEE Transactions on Electron Devices.

[6]  Pritish Narayanan,et al.  Neuromorphic computing using non-volatile memory , 2017 .

[7]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[8]  Weisong Shi,et al.  Edge Computing: Vision and Challenges , 2016, IEEE Internet of Things Journal.

[9]  H.-S. Philip Wong,et al.  Face classification using electronic synapses , 2017, Nature Communications.

[10]  Gökmen Tayfun,et al.  Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations , 2016, Front. Neurosci..

[11]  I-Ting Wang,et al.  3D Ta/TaOx/TiO2/Ti synaptic array and linearity tuning of weight update for hardware neural network applications , 2016, Nanotechnology.

[12]  H. Hwang,et al.  Optimized Programming Scheme Enabling Linear Potentiation in Filamentary HfO2 RRAM Synapse for Neuromorphic Systems , 2016, IEEE Transactions on Electron Devices.

[13]  Mahadev Satyanarayanan,et al.  The Emergence of Edge Computing , 2017, Computer.

[14]  Kuldip K. Paliwal,et al.  Bidirectional recurrent neural networks , 1997, IEEE Trans. Signal Process..

[15]  Jaeha Kim,et al.  Demonstration of Unsupervised Learning With Spike-Timing-Dependent Plasticity Using a TFT-Type NOR Flash Memory Array , 2018, IEEE Transactions on Electron Devices.

[16]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[17]  G. W. Burr,et al.  Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element , 2015, 2014 IEEE International Electron Devices Meeting.

[18]  Joel Emer,et al.  Eyeriss: an Energy-efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks Accessed Terms of Use , 2022 .

[19]  Paolo Fantini,et al.  Unsupervised Learning by Spike Timing Dependent Plasticity in Phase Change Memory (PCM) Synapses , 2016, Front. Neurosci..