Minimum description length neural networks for time series prediction.

Artificial neural networks (ANN) are typically composed of a large number of nonlinear functions (neurons) each with several linear and nonlinear parameters that are fitted to data through a computationally intensive training process. Longer training results in a closer fit to the data, but excessive training will lead to overfitting. We propose an alternative scheme that has previously been described for radial basis functions (RBF). We show that fundamental differences between ANN and RBF make application of this scheme to ANN nontrivial. Under this scheme, the training process is replaced by an optimal fitting routine, and overfitting is avoided by controlling the number of neurons in the network. We show that for time series modeling and prediction, this procedure leads to small models (few neurons) that mimic the underlying dynamics of the system well and do not overfit the data. We apply this algorithm to several computational and real systems including chaotic differential equations, the annual sunspot count, and experimental data obtained from a chaotic laser. Our experiments indicate that the structural differences between ANN and RBF make ANN particularly well suited to modeling chaotic time series data.

[1]  Michael Small,et al.  Temporal Evolution of nonlinear Dynamics in Ventricular Arrhythmia , 2001, Int. J. Bifurc. Chaos.

[2]  William H. Press,et al.  Numerical recipes in C , 2002 .

[3]  Howell Tong,et al.  Non-Linear Time Series , 1990 .

[4]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[5]  Jukka Saarinen,et al.  A network of autoregressive processing units for time series modeling , 1996 .

[6]  Michael Small,et al.  Correlation dimension: a pivotal statistic for non-constrained realizations of composite hypotheses in surrogate data analysis , 1998 .

[7]  H. Tong,et al.  Data transformation and self-exciting threshold autoregression , 1981 .

[8]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction , 2001 .

[9]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[10]  Andreas S. Weigend,et al.  Time Series Prediction: Forecasting the Future and Understanding the Past , 1994 .

[11]  Kevin Judd,et al.  Modeling continuous processes from data. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[12]  Stefano Boccaletti.  Space-time chaos : characterization, control and synchronization : Pamplona, Spain 19-23 June 2000 , 2001 .

[13]  Henry Leung,et al.  Prediction of noisy chaotic time series using an optimal radial basis function neural network , 2001, IEEE Trans. Neural Networks.

[14]  Nikolai F. Rulkov,et al.  Modeling and synchronizing chaotic systems from experimental data , 1994 .

[15]  Kevin Judd,et al.  Embedding as a modeling problem , 1998 .

[16]  Jukka Saarinen,et al.  Predictive Minimum Description Length Criterion for Time Series Modeling with Neural Networks , 1996, Neural Computation.

[17]  Horst Bischof,et al.  An efficient MDL-based construction of RBF networks , 1998, Neural Networks.

[18]  Michael C. Mozer,et al.  Mathematical Perspectives on Neural Networks , 1996 .

[19]  S. Ellacott,et al.  Neural networks : deterministic methods of analysis , 1996 .

[20]  H. Akaike.  A new look at the statistical model identification , 1974 .

[21]  Michael Small,et al.  Comparisons of new nonlinear modeling techniques with applications to infant respiration , 1998 .

[22]  Jorma Rissanen,et al.  Stochastic Complexity in Statistical Inquiry , 1989, World Scientific Series in Computer Science.

[23]  Diks,et al.  Efficient implementation of the gaussian kernel algorithm in estimating invariants and noise level from noisy time series data , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[24]  David B. Fogel.  An information criterion for optimal neural network selection , 1991, IEEE Trans. Neural Networks.

[25]  A. Mees,et al.  On selecting models for nonlinear time series , 1995 .

[26]  M Small,et al.  Is breathing in infants chaotic? Dimension estimates for respiratory patterns during quiet sleep. , 1999, Journal of applied physiology.