Memristor-based LSTM network with in situ training and its applications

Artificial neural networks (ANNs), such as the convolutional neural network (CNN) and long short-term memory (LSTM), have high complexity and contain large numbers of parameters. Memristor-based neural networks, which have the ability of in-memory and parallel computing, are therefore proposed to accelerate the operations of ANNs. In this paper, a memristor-based hardware realization of long short-term memory (LSTM) network with in situ training is presented. The designed memristor-based LSTM (MbLSTM) network is composed of memristor-based LSTM cell and memristor-based dense layer. Sigmoid and tanh (hyperbolic tangent) activation functions are approximately implemented through intentionally designing circuit parameters. A weight update scheme with row-parallel characteristic is put forward to update the conductance of memristors in crossbars. The highlights of MbLSTM include an effective hardware-based inference process and in situ training. The validity of MbLSTM is substantiated through classification tasks. The robustness of MbLSTM to conductance variations is also analyzed.

[1]  Wei Yang Lu,et al.  Nanoscale memristor device as synapse in neuromorphic systems. , 2010, Nano letters.

[2]  X. Miao,et al.  Activity-Dependent Synaptic Plasticity of a Chalcogenide Electronic Synapse for Neuromorphic Systems , 2014, Scientific Reports.

[3]  L. Chua.  Memristor-The missing circuit element , 1971 .

[4]  C.-J. Richard Shi,et al.  OCEAN: An On-Chip Incremental-Learning Enhanced Artificial Neural Network Processor With Multiple Gated-Recurrent-Unit Accelerators , 2018, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[5]  Eby G. Friedman,et al.  VTEAM – A General Model for Voltage Controlled Memristors , 2014 .

[6]  Fabien Alibart,et al.  Pattern classification by memristive crossbar circuits using ex situ and in situ training , 2013, Nature Communications.

[7]  Berin Martini,et al.  Recurrent Neural Networks Hardware Implementation on FPGA , 2015, ArXiv.

[8]  Mauro Forti,et al.  Memristor standard cellular neural networks computing in the flux-charge domain , 2017, Neural Networks.

[9]  Houshang Darabi,et al.  Multivariate LSTM-FCNs for Time Series Classification , 2018, Neural Networks.

[10]  Michael C. Mozer,et al.  A Focused Backpropagation Algorithm for Temporal Pattern Recognition , 1989, Complex Syst..

[11]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[12]  Alex Pappachen James,et al.  A memristor-based long short term memory circuit , 2018 .

[13]  Leon O. Chua,et al.  Everything You Wish to Know About Memristors But Are Afraid to Ask , 2015 .

[14]  PAUL J. WERBOS,et al.  Generalization of backpropagation with application to a recurrent gas market model , 1988, Neural Networks.

[15]  Chunwei Tian,et al.  Image denoising using deep CNN with batch renormalization , 2020, Neural Networks.

[16]  Yin Yang,et al.  Memristor-Based Echo State Network With Online Least Mean Square , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[17]  Jun Wu,et al.  3D Convolutional Neural Network based on memristor for video recognition , 2020, Pattern Recognit. Lett..

[18]  Bing J. Sheu,et al.  A high-precision VLSI winner-take-all circuit for self-organizing neural networks , 1993 .

[19]  Tayfun Gokmen,et al.  Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices , 2017, Front. Neurosci..

[20]  Kyeong-Sik Min,et al.  New Memristor-Based Crossbar Array Architecture with 50-% Area Reduction and 48-% Power Saving for Matrix-Vector Multiplication of Analog Neuromorphic Computing , 2014 .

[21]  Shuhui Li,et al.  An adaptive recurrent neural-network controller using a stabilization matrix and predictive inputs to solve a tracking problem under disturbances , 2014, Neural Networks.

[22]  Xiaoping Wang,et al.  A Novel Memristor-Based Circuit Implementation of Full-Function Pavlov Associative Memory Accorded With Biological Feature , 2018, IEEE Transactions on Circuits and Systems I: Regular Papers.

[23]  Alex Pappachen James,et al.  Learning in Memristive Neural Network Architectures Using Analog Backpropagation Circuits , 2018, IEEE Transactions on Circuits and Systems I: Regular Papers.

[24]  Qing Wu,et al.  Efficient and self-adaptive in-situ learning in multilayer memristor neural networks , 2018, Nature Communications.

[25]  Eby G. Friedman,et al.  Synaptic Characteristics of Ag/AgInSbTe/Ta-Based Memristor for Pattern Recognition Applications , 2017, IEEE Transactions on Electron Devices.

[26]  D. Stewart,et al.  The missing memristor found , 2008, Nature.

[27]  Shaolong Sun,et al.  Sparse Self-Attention LSTM for Sentiment Lexicon Construction , 2019, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[28]  Zhigang Zeng,et al.  Implementation of Memristive Neural Network With Full-Function Pavlov Associative Memory , 2016, IEEE Transactions on Circuits and Systems I: Regular Papers.

[29]  Ke Chen,et al.  Multi-label zero-shot human action recognition via joint latent ranking embedding , 2017, Neural Networks.

[30]  Zhigang Zeng,et al.  Memristor-based circuit implementation of pulse-coupled neural network with dynamical threshold generators , 2018, Neurocomputing.

[31]  Zhigang Zeng,et al.  Memristive LSTM Network for Sentiment Analysis , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[32]  Shiping Wen,et al.  A full-function Pavlov associative memory implementation with memristance changing circuit , 2018, Neurocomputing.

[33]  Qing Wu,et al.  Long short-term memory networks in memristor crossbar arrays , 2018, Nature Machine Intelligence.

[34]  Tayfun Gokmen,et al.  Training LSTM Networks With Resistive Cross-Point Devices , 2018, Front. Neurosci..

[35]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[36]  Zhigang Zeng,et al.  A memristor-based neural network circuit with synchronous weight adjustment , 2019, Neurocomputing.

[37]  Shuhui Li,et al.  Training Recurrent Neural Networks With the Levenberg–Marquardt Algorithm for Optimal Control of a Grid-Connected Converter , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[38]  Mikel Sanz,et al.  Perceptrons from Memristors , 2018, Neural Networks.

[39]  Zhigang Zeng,et al.  Memristive Fully Convolutional Network: An Accurate Hardware Image-Segmentor in Deep Learning , 2018, IEEE Transactions on Emerging Topics in Computational Intelligence.

[40]  Jürgen Schmidhuber,et al.  Learning to Forget: Continual Prediction with LSTM , 2000, Neural Computation.

[41]  Eby G. Friedman,et al.  Memristive Model for Synaptic Circuits , 2017, IEEE Transactions on Circuits and Systems II: Express Briefs.

[42]  H.-S. Philip Wong,et al.  Face classification using electronic synapses , 2017, Nature Communications.

[43]  Zhigang Zeng,et al.  General memristor with applications in multilayer neural networks , 2018, Neural Networks.

[44]  Eby G. Friedman,et al.  Memristor-Based Circuit Design for Multilayer Neural Networks , 2018, IEEE Transactions on Circuits and Systems I: Regular Papers.

[45]  Zhen-Hua Ling,et al.  A Sequential Neural Encoder With Latent Structured Description for Modeling Sentences , 2018, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[46]  Yiran Chen,et al.  Memristor Crossbar-Based Neuromorphic Computing System: A Case Study , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[47]  Shukai Duan,et al.  Memristive pulse coupled neural network with applications in medical image processing , 2017, Neurocomputing.

[48]  Leon O. Chua,et al.  Memristor Bridge Synapse-Based Neural Network and Its Learning , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[49]  Catherine E. Graves,et al.  Memristor‐Based Analog Computation and Neural Network Classification with a Dot Product Engine , 2018, Advanced materials.

[50]  Avinoam Kolodny,et al.  Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[51]  Uri C. Weiser,et al.  TEAM: ThrEshold Adaptive Memristor Model , 2013, IEEE Transactions on Circuits and Systems I: Regular Papers.

[52]  Chris Yakopcic,et al.  On-chip training of memristor crossbar based multi-layer neural networks , 2017, Microelectron. J..