Noninvertibility in neural networks

We present and discuss an inherent shortcoming of neural networks used as discrete-time models in system identification, time series processing, and prediction. Trajectories of nonlinear ordinary differential equations (ODEs) can, under reasonable assumptions, be integrated uniquely backward in time. Discrete-time neural network mappings derived from time series, on the other hand, can give rise to multiple trajectories when followed backward in time: they are in principle noninvertible. This fundamental difference can lead to model predictions that are not only slightly quantitatively different, but qualitatively inconsistent with continuous time series. We discuss how noninvertibility arises, present key analytical concepts and some of its phenomenology. Using two illustrative examples (one experimental and one computational), we demonstrate when noninvertibility becomes an important factor in the validity of artificial neural network (ANN) predictions, and show some of the overall complexity of the predicted pathological dynamical behavior. These concepts can be used to probe the validity of ANN time series models, as well as provide guidelines for the acquisition of additional training data.

[1]  A. Lapedes,et al.  Nonlinear signal processing using neural networks: Prediction and system modelling , 1987 .

[2]  Andreas S. Weigend,et al.  Time Series Prediction: Forecasting the Future and Understanding the Past , 1994 .

[3]  Dale E. Seborg,et al.  Nonlinear internal model control strategy for neural network models , 1992 .

[4]  Hong-Te Su,et al.  Identification of Chemical Processes using Recurrent Networks , 1991, 1991 American Control Conference.

[5]  Yaman Arkun,et al.  Study of the control-relevant properties of backpropagation neural network models of nonlinear dynamical systems , 1992 .

[6]  Ioannis G. Kevrekidis,et al.  Continuous time modeling of nonlinear systems: a neural network-based approach , 1993, IEEE International Conference on Neural Networks.

[7]  A. Lapedes,et al.  Global bifurcations in Rayleigh-Be´nard convection: experiments, empirical maps and numerical bifurcation analysis , 1993, comp-gas/9305004.

[8]  Raymond A. Adomaitis,et al.  Noninvertibility in neural networks , 1993, IEEE International Conference on Neural Networks.

[9]  Mark A. Kramer,et al.  Algorithm 658: ODESSA–an ordinary differential equation solver with explicit simultaneous sensitivity analysis , 1988, TOMS.

[10]  James P. Crutchfield,et al.  Geometry from a Time Series , 1980 .

[11]  Ioannis G. Kevrekidis,et al.  Nonlinear signal processing and system identification: applications to time series from electrochemical reactions , 1990 .

[12]  Laura Gardini.  Some global bifurcations of two-dimensional endomorphisms by use of critical lines , 1992 .

[13]  N.V. Bhat,et al.  Modeling chemical process systems via neural computation , 1990, IEEE Control Systems Magazine.

[14]  B. Ydstie.  Forecasting and control using adaptive connectionist networks , 1990 .

[15]  Phillippe Langonnet.  Process control with neural networks: an example , 1992, Defense, Security, and Sensing.

[16]  Rahmat A. Shoureshi,et al.  Neural networks for system identification , 1990 .

[17]  Katharina Krischer,et al.  Chaos and Interior Crisis in an Electrochemical Reaction , 1991 .

[18]  Oluseyi Olurotimi,et al.  Recurrent neural network training with feedforward complexity , 1994, IEEE Trans. Neural Networks.

[19]  I. Kevrekidis,et al.  Noninvertibility and resonance in discrete-time neural networks for time-series processing , 1998 .

[20]  Raymond A. Adomaitis,et al.  Noninvertibility and the structure of basins of attraction in a model adaptive control system , 1991 .

[21]  Christian Mira,et al.  Recurrences and Discrete Dynamic Systems , 1980 .

[22]  C. Robinson.  Dynamical Systems: Stability, Symbolic Dynamics, and Chaos , 1994 .

[23]  Ioannis G. Kevrekidis,et al.  DISCRETE- vs. CONTINUOUS-TIME NONLINEAR SIGNAL PROCESSING OF Cu ELECTRODISSOLUTION DATA , 1992 .

[24]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[25]  C Mira,et al.  Chaotic Dynamics: From the One-Dimensional Endomorphism to the Two-Dimensional Diffeomorphism , 1987 .

[26]  I.G. Kevrekidis,et al.  Continuous-time nonlinear signal processing: a neural network based approach for gray box identification , 1994, Proceedings of IEEE Workshop on Neural Networks for Signal Processing.

[27]  Rahmat Shoureshi,et al.  A Neural Network Approach for Identification of Continuous-Time Nonlinear Dynamic Systems , 1991, 1991 American Control Conference.

[28]  Rutherford Aris,et al.  Rate multiplicity and oscillations in single species surface reactions , 1984 .

[29]  F. Takens.  Detecting strange attractors in turbulence , 1981 .

[30]  Graham C. Goodwin,et al.  Adaptive filtering prediction and control , 1984 .

[31]  Christian Mira,et al.  On Some Properties of Invariant Sets of Two-Dimensional Noninvertible Maps , 1997 .

[32]  Stephen A. Billings,et al.  Non-linear system identification using neural networks , 1990 .

[33]  Edward N. Lorenz,et al.  Computational chaos-a prelude to computational instability , 1989 .