Time series modelling with recurrent CBP

We address the construction of recurrent neural networks by the use of constructive backpropagation (CBP). The benefits of the proposed scheme include: 1) fully recurrent networks with arbitrary number of layers can be constructed efficiently; and 2) after the network has been constructed one can continue the adaptation of the network weights as well as continue structure adaptation. This includes both addition and deletion of neurons/layers in a computationally efficient manner. Thus the investigated method is very flexible compared to many previous methods. In addition, according to our time series prediction experiments, the proposed method is competitive compared to the well known recurrent cascade-correlation method.

[1]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[2]  C. Lee Giles,et al.  Constructive learning of recurrent neural networks: limitations of recurrent cascade correlation and a simple solution , 1995, IEEE Trans. Neural Networks.

[3]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[4]  Marwan A. Jabri,et al.  Constructive Learning Using Internal Representation Conflicts , 1993, NIPS.

[5]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[6]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[7]  Dragan Obradovic,et al.  On-line training of recurrent neural networks with continuous topology adaptation , 1996, IEEE Trans. Neural Networks.

[8]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[9]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[10]  Jacek M. Zurada,et al.  Sequential network construction for time series prediction , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[11]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[12]  J. Thompson,et al.  Nonlinear Dynamics and Chaos: Geometrical Methods for Engineers and Scientists , 1986 .

[13]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.