论文信息 - On the weight dynamics of recurrent learning

On the weight dynamics of recurrent learning

We derive continuous-time batch and online versions of the recently introduced efficient O(N 2 ) training algorithm of Atiya and Parlos (2000) for fully recurrent networks. A mathematical analysis of the respective weight dy- namics yields that efficient learning is achieved although relative rates of weight change remain constant due to the way errors are backpropagated. The result is a highly structured network where an unspecific internal dynamical reservoir can be distinguished from the output layer, which learns faster and changes at much higher rates. We discuss this result with respect to the recently introduced "echo state" and "liquid state" networks, which have similar structure.

Jochen J. Steil | Ulf D. Schiller

[1] H. Ritter,et al. Recurrent learning of input-output stable behaviour in function space: A case study with the Roessler attractor , 1999 .

[2] Barak A. Pearlmutter. Gradient calculations for dynamic recurrent neural networks: a survey , 1995, IEEE Trans. Neural Networks.

[3] Henry Markram,et al. The "Liquid Computer": A Novel Strategy for Real-Time Computing on Time Series , 2002 .

[4] Jochen J. Steil,et al. Perspectives on learning with recurrent neural networks , 2002, ESANN.

[5] Herbert Jaeger,et al. Adaptive Nonlinear System Identification with Echo State Networks , 2002, NIPS.

[6] Herbert Jaeger,et al. The''echo state''approach to analysing and training recurrent neural networks , 2001 .

[7] Amir F. Atiya,et al. New results on recurrent network training: unifying the algorithms and accelerating convergence , 2000, IEEE Trans. Neural Networks Learn. Syst..