Training Neural Networks using Memristive Devices with Nonlinear Accumulative Behavior

Memristive devices when organized in crossbar arrays can be used to accelerate the training of deep neural networks through in-memory computing. In this paper, we propose a scheme that addresses a key challenge, namely, the non-linear accumulative behavior of memristive devices. Data entering and leaving the memristive crossbar array is pre- and post-processed to change the effective mapping of weight matrices onto the conductance matrix. The impact of the proposed scheme is studied for training neural networks using phase-change memory (PCM) devices as synaptic elements. The scheme is shown to significantly improve the network's classification accuracy, allowing us to reach the performance of an ideal linear device.

[1]  Tayfun Gokmen,et al.  Training LSTM Networks With Resistive Cross-Point Devices , 2018, Front. Neurosci..

[2]  Abu Sebastian,et al.  Tutorial: Brain-inspired computing using phase-change memory devices , 2018, Journal of Applied Physics.

[3]  E. Eleftheriou,et al.  A phase-change memory model for neuromorphic computing , 2018, Journal of Applied Physics.

[4]  Shahram Minaei,et al.  Applications of a CMOS current squaring circuit in analog signal processing , 2015, 2015 38th International Conference on Telecommunications and Signal Processing (TSP).

[5]  Evangelos Eleftheriou,et al.  Mixed-precision architecture based on computational memory for training deep neural networks , 2018, 2018 IEEE International Symposium on Circuits and Systems (ISCAS).

[6]  Shimeng Yu,et al.  Synaptic electronics: materials, devices and applications , 2013, Nanotechnology.

[7]  M. Breitwisch,et al.  Novel Lithography-Independent Pore Phase Change Memory , 2007, 2007 IEEE Symposium on VLSI Technology.

[8]  Heiner Giefers,et al.  Compressed Sensing With Approximate Message Passing Using In-Memory Computing , 2018, IEEE Transactions on Electron Devices.

[9]  Catherine E. Graves,et al.  Memristor‐Based Analog Computation and Neural Network Classification with a Dot Product Engine , 2018, Advanced materials.

[10]  Wei D. Lu,et al.  Sparse coding with memristor networks. , 2017, Nature nanotechnology.

[11]  Pritish Narayanan,et al.  Equivalent-accuracy accelerated neural-network training using analogue memory , 2018, Nature.

[12]  Pritish Narayanan,et al.  Experimental Demonstration and Tolerancing of a Large-Scale Neural Network (165 000 Synapses) Using Phase-Change Memory as the Synaptic Weight Element , 2014, IEEE Transactions on Electron Devices.

[13]  Damien Querlioz,et al.  Narrow Heater Bottom Electrode‐Based Phase Change Memory as a Bidirectional Artificial Synapse , 2018, Advanced Electronic Materials.

[14]  Abu Sebastian,et al.  Accumulation-Based Computing Using Phase-Change Memories With FET Access Devices , 2015, IEEE Electron Device Letters.

[15]  Gökmen Tayfun,et al.  Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations , 2016, Front. Neurosci..

[16]  G. W. Burr,et al.  Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element , 2015, 2014 IEEE International Electron Devices Meeting.

[17]  Heiner Giefers,et al.  Mixed-precision in-memory computing , 2017, Nature Electronics.

[18]  H.-S. Philip Wong,et al.  In-memory computing with resistive switching devices , 2018, Nature Electronics.

[19]  Thomas P. Parnell,et al.  Temporal correlation detection using computational phase-change memory , 2017, Nature Communications.

[20]  N Gong,et al.  Signal and noise extraction from analog memory elements for neuromorphic computing , 2018, Nature Communications.

[21]  Daniel Krebs,et al.  Crystal growth within a phase change memory cell , 2014, Nature Communications.

[22]  Manuel Le Gallo,et al.  Stochastic phase-change neurons. , 2016, Nature nanotechnology.

[23]  Yusuf Leblebici,et al.  Neuromorphic computing with multi-memristive synapses , 2017, Nature Communications.

[24]  Hyunsang Hwang,et al.  Improved Synaptic Behavior of CBRAM Using Internal Voltage Divider for Neuromorphic Systems , 2018, IEEE Transactions on Electron Devices.

[25]  Pritish Narayanan,et al.  Neuromorphic computing using non-volatile memory , 2017 .

[26]  Tayfun Gokmen,et al.  Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices , 2017, Front. Neurosci..

[27]  Evangelos Eleftheriou,et al.  Inherent stochasticity in phase-change memory devices , 2016, 2016 46th European Solid-State Device Research Conference (ESSDERC).