A Normalized Adaptive Training of Recurrent Neural Networks With Augmented Error Gradient

For training algorithms of recurrent neural networks (RNN), convergent speed and training error are always two contradictory performances. In this letter, we propose a normalized adaptive recurrent learning (NARL) to obtain a tradeoff between transient and steady-state response. An augmented term is added to error gradient to exactly model the derivative of cost function with respect to hidden layer weight. The influence of the induced gain of activation function on training stability is also taken into consideration. Moreover, adaptive learning rate is employed to improve the robustness of the gradient training. Finally, computer simulations of a model prediction problem are synthesized to give comparisons between NARL and conventional normalized real-time recurrent learning (N-RTRL).

[1]  Lars-Henning Zetterberg,et al.  Identification of certain time-varying nonlinear Wiener and Hammerstein systems , 2001, IEEE Trans. Signal Process..

[2]  Kishan G. Mehrotra,et al.  Characterization of a Class of Sigmoid Functions with Applications to Neural Networks , 1996, Neural Networks.

[3]  Anke Meyer-Bäse,et al.  Local exponential stability of competitive neural networks with different time scales , 2004, Eng. Appl. Artif. Intell..

[4]  Anke Meyer-Bäse,et al.  Singular Perturbation Analysis of Competitive Neural Networks with Different Time Scales , 1996, Neural Computation.

[5]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[6]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[7]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction , 2001 .

[8]  A. Meyer-Base,et al.  Flow invariance for competitive multi-modal neural networks , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[9]  Danilo P. Mandic,et al.  Data-Reusing Recurrent Neural Adaptive Filters , 2002, Neural Computation.

[10]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[11]  Gail D. Baura,et al.  Nonlinear System Identification , 2002 .

[12]  A. Rodriguez-Vazquez,et al.  Modular analog continuous-time VLSI neural networks with on-chip Hebbian learning and analog storage , 1992, [Proceedings] 1992 IEEE International Symposium on Circuits and Systems.

[13]  C. Malsburg,et al.  How patterned neural connections can be set up by self-organization , 1976, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[14]  Man-Wai Mak,et al.  On the improvement of the real time recurrent learning algorithm for recurrent neural networks , 1999, Neurocomputing.

[15]  O. Nelles.  Nonlinear System Identification , 2001 .

[16]  Fuchun Sun,et al.  Neural network-based adaptive controller design of robotic manipulators with an observer , 2001, IEEE Trans. Neural Networks.

[17]  Ali H. Sayed,et al.  Supervised learning of perceptron and output feedback dynamic networks: a feedback analysis via the small gain theorem , 1997, IEEE Trans. Neural Networks.

[18]  Anke Meyer-Bäse,et al.  Global exponential stability of competitive neural networks with different time scales , 2003, IEEE Trans. Neural Networks.

[19]  Ah Chung Tsoi,et al.  Locally recurrent globally feedforward networks: a critical review of architectures , 1994, IEEE Trans. Neural Networks.

[20]  C. A. Desoer,et al.  Nonlinear Systems Analysis , 1978 .

[21]  XuLei Yang,et al.  Robust Recurrent Neural Network Control of Biped Robot , 2007, J. Intell. Robotic Syst..

[22]  Helge J. Ritter,et al.  Self-organizing maps: Local competition and evolutionary optimization , 1994, Neural Networks.

[23]  B. V. K. Vijaya Kumar,et al.  Emulating the dynamics for a class of laterally inhibited neural networks , 1989, Neural Networks.

[24]  Danilo P. Mandic,et al.  A normalised real time recurrent learning algorithm , 2000, Signal Process..

[25]  Ali H. Sayed,et al.  A time-domain feedback analysis of filtered-error adaptive gradient algorithms , 1996, IEEE Trans. Signal Process..

[26]  Jing Peng,et al.  An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories , 1990, Neural Computation.

[27]  Qing Song,et al.  Robust backpropagation training algorithm for multilayered neural tracking controller , 1999, IEEE Trans. Neural Networks.