Fast Training of Deep LSTM Networks

Deep recurrent neural networks (RNN), such as LSTM, have many advantages over forward networks. However, the LSTM training method, such as backward propagation through time (BPTT), is really slow.

[1]  Steve B. Jiang,et al.  Nonlinear Systems Identification Using Deep Dynamic Neural Networks , 2016, ArXiv.

[2]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[3]  Walter W. Focke,et al.  Mixture Models Based on Neural Network Averaging , 2006, Neural Computation.

[4]  Johan A. K. Suykens,et al.  Wiener-Hammerstein Benchmark , 2009 .

[5]  George E. P. Box,et al.  Time Series Analysis: Box/Time Series Analysis , 2008 .

[6]  Xizhao Wang,et al.  Advances in neural network based learning , 2014, Int. J. Mach. Learn. Cybern..

[7]  Liang Wang,et al.  Complex systems modeling via fuzzy logic , 1996, IEEE Trans. Syst. Man Cybern. Part B.

[8]  Wen Yu,et al.  Nonlinear system identification using discrete-time recurrent neural networks with stable learning algorithms , 2004, Inf. Sci..

[9]  Xizhao Wang,et al.  A review on neural networks with random weights , 2018, Neurocomputing.

[10]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.

[11]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[12]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[13]  Wen Yu,et al.  Discrete-time neuro identification without robust modification , 2003 .

[14]  Amit Kumar Yadav,et al.  Solar radiation prediction using Artificial Neural Network techniques: A review , 2014 .

[15]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[16]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[17]  Yu Wang,et al.  A new concept using LSTM Neural Networks for dynamic system identification , 2017, 2017 American Control Conference (ACC).

[18]  Xizhao Wang,et al.  Non-iterative approaches in training feed-forward neural networks and their applications , 2018, Soft Computing.

[19]  Wen Yu,et al.  Recurrent Neural Networks Training With Stable Bounding Ellipsoid Algorithm , 2009, IEEE Transactions on Neural Networks.

[20]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[21]  Noriaki Hirose,et al.  Modeling of rolling friction by recurrent neural network using LSTM , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[22]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[23]  O. Nelles.  Nonlinear System Identification: From Classical Approaches to Neural Networks and Fuzzy Models , 2000 .

[24]  Kumpati S. Narendra,et al.  Gradient methods for the optimization of dynamical systems containing neural networks , 1991, IEEE Trans. Neural Networks.

[25]  Michio Sugeno,et al.  A fuzzy-logic-based approach to qualitative modeling , 1993, IEEE Trans. Fuzzy Syst..