Implementation of Dropout Neuronal Units Based on Stochastic Memristive Devices in Neural Networks with High Classification Accuracy

Abstract Neural networks based on memristive devices have achieved great progress recently. However, memristive synapses with nonlinearity and asymmetry seriously limit the classification accuracy. Moreover, insufficient number of training samples in many cases also have negative effect on the classification accuracy of neural networks due to overfitting. In this work, dropout neuronal units are developed based on stochastic volatile memristive devices of Ag/Ta2O5:Ag/Pt. The memristive neural network using the dropout neuronal units effectively solves the problem of overfitting and mitigates the negative effects of the nonideality of memristive synapses, eventually achieves a classification accuracy comparable to the theoretical limit. The stochastic and volatile switching performances of the Ag/Ta2O5:Ag/Pt device are attributed to the stochastical rupture of the Ag filament under high electrical stress in the Ta2O5 layer, according to the TEM observation and the kinetic Monte Carlo simulation.

[1]  Hangbing Lv,et al.  An Artificial Neuron Based on a Threshold Switching Memristor , 2018, IEEE Electron Device Letters.

[2]  Yi Li,et al.  Alleviating Conductance Nonlinearity via Pulse Shape Designs in TaOx Memristive Synapses , 2019, IEEE Transactions on Electron Devices.

[3]  Qing Wu,et al.  Efficient and self-adaptive in-situ learning in multilayer memristor neural networks , 2018, Nature Communications.

[4]  Wei D. Lu,et al.  Electrochemical dynamics of nanoscale metallic inclusions in dielectrics , 2014, Nature Communications.

[5]  John Paul Strachan,et al.  Chaotic dynamics in nanoscale NbO2 Mott memristors for analogue computing , 2017, Nature.

[6]  Bin Gao,et al.  Fully hardware-implemented memristor convolutional neural network , 2020, Nature.

[7]  Chih-Cheng Chang,et al.  Mitigating Asymmetric Nonlinear Weight Update Effects in Hardware Neural Network Based on Analog Resistive Synapse , 2018, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[8]  Rui Yang,et al.  Memristive Synapses and Neurons for Bioinspired Computing , 2019, Advanced Electronic Materials.

[9]  Ali Khiat,et al.  Unsupervised learning in probabilistic neural networks with multi-state metal-oxide memristive synapses , 2016, Nature Communications.

[10]  Boudewijn F. van Dongen,et al.  Process mining: a two-step approach to balance between underfitting and overfitting , 2008, Software & Systems Modeling.

[11]  Peng Lin,et al.  Fully memristive neural networks for pattern classification with unsupervised learning , 2018 .

[12]  Xue-Bing Yin,et al.  Pt/WO3/FTO memristive devices with recoverable pseudo-electroforming for time-delay switches in neuromorphic computing. , 2016, Physical chemistry chemical physics : PCCP.

[13]  M. Pickett,et al.  A scalable neuristor built with Mott memristors. , 2013, Nature materials.

[14]  H.-S. Philip Wong,et al.  Face classification using electronic synapses , 2017, Nature Communications.

[15]  Jan van den Hurk,et al.  Nanobatteries in redox-based resistive switches require extension of memristor theory , 2013, Nature Communications.

[16]  Young Sun,et al.  All‐Solid‐State Synaptic Transistor with Ultralow Conductance for Neuromorphic Computing , 2018, Advanced Functional Materials.

[17]  David-Wei Zhang,et al.  An Organic Flexible Artificial Bio-Synapses with Long-Term Plasticity for Neuromorphic Computing , 2018, Micromachines.

[18]  Qi Liu,et al.  Eliminating Negative‐SET Behavior by Suppressing Nanofilament Overgrowth in Cation‐Based Memory , 2016, Advanced materials.

[19]  Rainer Waser,et al.  Understanding filamentary growth in electrochemical metallization memory cells using kinetic Monte Carlo simulations. , 2015, Nanoscale.

[20]  H. Hwang,et al.  Improved Synaptic Behavior Under Identical Pulses Using AlOx/HfO2 Bilayer RRAM Array for Neuromorphic Systems , 2016, IEEE Electron Device Letters.

[21]  M. Rozenberg,et al.  A Leaky‐Integrate‐and‐Fire Neuron Analog Realized with a Mott Insulator , 2017 .

[22]  Xin Guo,et al.  Coexistence of analog and digital resistive switching in BiFeO3-based memristive devices , 2016 .

[23]  Chun-Wei Huang,et al.  Direct Observation of Dual-Filament Switching Behaviors in Ta2 O5 -Based Memristors. , 2017, Small.

[24]  Jue Xiong,et al.  Quasi‐Hodgkin–Huxley Neurons with Leaky Integrate‐and‐Fire Functions Physically Realized with Memristive Devices , 2018, Advanced materials.

[25]  Zhengya Zhang,et al.  A fully integrated reprogrammable memristor–CMOS system for efficient multiply–accumulate operations , 2019, Nature Electronics.

[26]  Manuel Le Gallo,et al.  Stochastic phase-change neurons. , 2016, Nature nanotechnology.

[27]  Wei D. Lu,et al.  Experimental Demonstration of Feature Extraction and Dimensionality Reduction Using Memristor Networks. , 2017, Nano letters.

[28]  Field-driven migration of bipolar metal particles on solid electrolytes , 2008 .

[29]  Qi Liu,et al.  Direct Observation of Conversion Between Threshold Switching and Memory Switching Induced by Conductive Filament Morphology , 2014 .

[30]  Jan Trieschmann,et al.  Kinetic Simulation of Filament Growth Dynamics in Memristive Electrochemical Metallization Devices , 2015, 1509.00208.

[31]  I. Valov,et al.  Effects of moisture and redox reactions in VCM and ECM resistive switching memories , 2018, Journal of Physics D: Applied Physics.

[32]  Qi Liu,et al.  Transformation of threshold volatile switching to quantum point contact originated nonvolatile switching in graphene interface controlled memory devices , 2019, Nanoscale advances.

[33]  X. D. Gao,et al.  Stable bipolar resistance switching behaviour induced by a soft breakdown process at the Al/La0.7Ca0.3MnO3 interface , 2009 .

[34]  Xue-li Li,et al.  Endurance improvement of resistance switching behaviors in the La0.7Ca0.3MnO3 film based devices with Ag–Al alloy top electrodes , 2010 .

[35]  Qing Wu,et al.  Long short-term memory networks in memristor crossbar arrays , 2018, Nature Machine Intelligence.

[36]  Qing Wu,et al.  A novel true random number generator based on a stochastic diffusive memristor , 2017, Nature Communications.

[37]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[38]  Xiaomin Li,et al.  Improvement of resistance switching properties for metal/La0.7Ca0.3MnO3/Pt devices , 2011 .

[39]  J. Yang,et al.  Sub-10 nm Ta Channel Responsible for Superior Performance of a HfO2 Memristor , 2016, Scientific Reports.

[40]  J. Yang,et al.  Memristors with diffusive dynamics as synaptic emulators for neuromorphic computing. , 2017, Nature materials.

[41]  M. Marinella,et al.  A non-volatile organic electrochemical device as a low-voltage artificial synapse for neuromorphic computing. , 2017, Nature materials.

[42]  Peng Lin,et al.  Reinforcement learning with analogue memristor arrays , 2019, Nature Electronics.

[43]  Shimeng Yu,et al.  Fully parallel write/read in resistive synaptic array for accelerating on-chip learning , 2015, Nanotechnology.

[44]  Sangheon Oh,et al.  Neuroinspired unsupervised learning and pruning with subquantum CBRAM arrays , 2018, Nature Communications.

[45]  Catherine E. Graves,et al.  Memristor‐Based Analog Computation and Neural Network Classification with a Dot Product Engine , 2018, Advanced materials.