Two constructive algorithms for improved time series processing with recurrent neural networks

Because of their universal approximation capabilities, recurrent neural networks are an attractive choice for building models of time series out of available data. Medium- and long-term dependencies are easier to learn when the recurrent network contains time-delayed connections. We propose two constructive algorithms which are able to choose the right locations and delays of such connections. To evaluate the capabilities of these algorithms, we use both natural data and synthetic data having built-in time delays. We then compare the two algorithms in order to define their domain of interest. The results we obtain on several benchmarks show that, by selectively adding a few time-delayed connections to recurrent networks, one is able to improve upon the results reported in the literature, while using significantly fewer parameters.

[1]  Donald E. Waagen,et al.  Evolving recurrent perceptrons for time-series modeling , 1994, IEEE Trans. Neural Networks.

[2]  Richard J. Duro,et al.  Discrete-time backpropagation for training synaptic delay-based artificial neural networks , 1999, IEEE Trans. Neural Networks.

[3]  Ah Chung Tsoi,et al.  A unifying view of some training algorithms for multilayer perceptrons with FIR filter synapses , 1994, Proceedings of IEEE Workshop on Neural Networks for Signal Processing.

[4]  C. L. Giles,et al.  Inserting rules into recurrent neural networks , 1992, Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop.

[5]  Gwilym M. Jenkins,et al.  Time series analysis, forecasting and control , 1971 .

[6]  Martin Casdagli,et al.  Nonlinear prediction of chaotic time series , 1989 .

[7]  Emanuel Marom,et al.  Efficient Training of Recurrent Neural Network with Time Delays , 1997, Neural Networks.

[8]  Yoshua Bengio,et al.  Hierarchical Recurrent Neural Networks for Long-Term Dependencies , 1995, NIPS.

[9]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[10]  H. Tong,et al.  Threshold Autoregression, Limit Cycles and Cyclical Data , 1980 .

[11]  Peter Tiño,et al.  Learning long-term dependencies in NARX recurrent neural networks , 1996, IEEE Trans. Neural Networks.

[12]  M. Hénon,et al.  A two-dimensional mapping with a strange attractor , 1976 .

[13]  P. Gallinari,et al.  Recurrent Neural Networks with Delays , 1994 .

[14]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[15]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[16]  David E. Rumelhart,et al.  Predicting the Future: a Connectionist Approach , 1990, Int. J. Neural Syst..

[17]  J. Doyne Farmer,et al.  Exploiting Chaos to Predict the Future and Reduce Noise , 1989 .