On the training of recurrent neural networks

This paper proposes a new approach which combines unsupervised and supervised learning for training recurrent neural networks (RNNs). In this approach, the weights between input and hidden layers were determined according to an unsupervised procedure relying on the Kohonen algorithm and the weights between hidden and output layers were updated according to a supervised procedure based on dynamic gradient descent method. The simulation results show that the proposed method performs well in comparison with the back propagation through time algorithm.

[1]  Gwilym M. Jenkins,et al.  Time series analysis, forecasting and control , 1972 .

[2]  Mohamed Chtourou,et al.  A Hybrid Training Algorithm for Feedforward Neural Networks , 2006, Neural Processing Letters.

[3]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[4]  Mohamed Chtourou,et al.  A hybrid approach for training recurrent neural networks: application to multi-step-ahead prediction of noisy and large data sets , 2007, Neural Computing and Applications.

[5]  Yeng Chai Soh,et al.  Robust Adaptive Gradient-Descent Training Algorithm for Recurrent Neural Networks in Discrete Time Domain , 2008, IEEE Transactions on Neural Networks.

[6]  Witold Pedrycz,et al.  Fuzzy prediction architecture using recurrent neural networks , 2009, Neurocomputing.

[7]  G. V. Puskorius,et al.  A signal processing framework based on dynamic neural networks with application to problems in adaptation, filtering, and classification , 1998, Proc. IEEE.

[8]  Jacob Barhen,et al.  Learning a trajectory using adjoint functions and teacher forcing , 1992, Neural Networks.

[9]  Lee A. Feldkamp,et al.  Recurrent network training with the decoupled-extended-Kalman-filter algorithm , 1992, Defense, Security, and Sensing.

[10]  Elif Derya Übeyli,et al.  A recurrent neural network classifier for Doppler ultrasound blood flow signals , 2006, Pattern Recognit. Lett..

[11]  Aniket A. Vartak,et al.  On-line Gauss-Newton-based learning for fully recurrent neural networks , 2005 .

[12]  Xiaoou Li,et al.  Dynamic system identification via recurrent multilayer perceptron , 2002, Inf. Sci..

[13]  Richard D. Braatz,et al.  On the "Identification and control of dynamical systems using neural networks" , 1997, IEEE Trans. Neural Networks.

[14]  Martin T. Hagan,et al.  Backpropagation Algorithms for a Broad Class of Dynamic Networks , 2007, IEEE Transactions on Neural Networks.

[15]  Teuvo Kohonen,et al.  The self-organizing map , 1990 .

[16]  Pierre Roussel-Ragot,et al.  Training recurrent neural networks: why and how? An illustration in dynamical process modeling , 1994, IEEE Trans. Neural Networks.

[17]  Ronald J. Williams,et al.  Training recurrent networks using the extended Kalman filter , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[18]  Amir F. Atiya,et al.  New results on recurrent network training: unifying the algorithms and accelerating convergence , 2000, IEEE Trans. Neural Networks Learn. Syst..

[19]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[20]  P J Webros.  BACKPROPAGATION THROUGH TIME: WHAT IT DOES AND HOW TO DO IT , 1990 .

[21]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[22]  Mohamed Chtourou,et al.  A fuzzy neighborhood-based training algorithm for feedforward neural networks , 2007, Neural Computing and Applications.

[23]  Les E. Atlas,et al.  Recurrent neural networks and robust time series prediction , 1994, IEEE Trans. Neural Networks.

[24]  Jacob Barhen,et al.  Adjoint-Functions and Temporal Learning Algorithms in Neural Networks , 1990, NIPS.

[25]  David R. Cox,et al.  Time Series Analysis , 2012 .

[26]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[27]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction , 2001 .

[28]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.