Understanding State Space Organization in Recurrent Neural Networks with Iterative Function Systems Dynamics

We study a novel recurrent network architecture with dynamics of iterative function systems used in chaos game representations of DNA sequences [16, 11] We show that such networks code the temporal and statistical structure of input sequences in a strict mathematical sense: generalized dimensions of network states are in direct correspondence with statistical properties of input sequences expressed via generalized Renyi entropy spectra. We also argue and experimentally illustrate that the commonly used heuristic of finite state machine extraction by network state space quantization corresponds in this case to variable memory length Markov model construction.

[1]  Omid Omidvar,et al.  Neural Networks and Pattern Recognition , 1997 .

[2]  J. McCauley.  Chaos, dynamics, and fractals : an algorithmic approach to deterministic chaos , 1993 .

[3]  V. Vojtek,et al.  Modeling Complex Symbolic Sequences with Neural Based Systems , 1997, ICANNGA.

[4]  R. Palmer,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[5]  Peter Tiño,et al.  Recurrent Neural Networks with Iterated Function Systems Dynamics , 1998, NC.

[6]  Peter Grassberger,et al.  Information and Complexity Measures in Dynamical Systems , 1991 .

[7]  H. J. Jeffrey.  Chaos game representation of gene structure. , 1990, Nucleic acids research.

[8]  Young,et al.  Inferring statistical complexity. , 1989, Physical review letters.

[9]  Dana Ron,et al.  The Power of Amnesia , 1993, NIPS.

[10]  Michael F. Barnsley,et al.  Fractals everywhere , 1988 .

[11]  Ramón Román-Roldán,et al.  Entropic feature for sequence pattern through iterated function systems , 1994, Pattern Recognit. Lett..

[12]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[13]  Panagiotis Manolios,et al.  First-Order Recurrent Neural Networks and Deterministic Finite State Automata , 1994, Neural Computation.

[14]  François E. Cellier,et al.  Artificial Neural Networks and Genetic Algorithms , 1991 .

[15]  C. Beck,et al.  Thermodynamics of chaotic systems , 1993 .

[16]  Mike Casey,et al.  The Dynamics of Discrete-Time Computation, with Application to Recurrent Neural Networks and Finite State Machine Extraction , 1996, Neural Computation.

[17]  Peter Tiño,et al.  Extracting finite-state representations from recurrent neural networks trained on chaotic symbolic sequences , 1999, IEEE Trans. Neural Networks.

[18]  Ebeling,et al.  Self-similar sequences and universal scaling of dynamical entropies. , 1996, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[19]  A. Rényi.  On the dimension and entropy of probability distributions , 1959 .

[20]  W. H. Zurek.  Complexity, Entropy and the Physics of Information , 1990 .

[21]  J. Oliver,et al.  Entropic profiles of DNA sequences through chaos-game-derived images. , 1993, Journal of theoretical biology.

[22]  Y. Peres,et al.  Measures of full dimension on affine-invariant sets , 1996, Ergodic Theory and Dynamical Systems.

[23]  C. Lee Giles,et al.  Extraction of rules from discrete-time recurrent neural networks , 1996, Neural Networks.

[24]  Peter Tiño,et al.  Learning and Extracting Initial Mealy Automata with a Modular Neural Network Model , 1995, Neural Comput..

[25]  Peter Tiño,et al.  Spatial representation of symbolic sequences through iterative function systems , 1999, IEEE Trans. Syst. Man Cybern. Part A.

[26]  Peter Tiňo,et al.  Finite State Machines and Recurrent Neural Networks -- Automata and Dynamical Systems Approaches , 1995 .

[27]  Gerald Sommer,et al.  Dynamic Cell Structure Learns Perfectly Topology Preserving Map , 1995, Neural Computation.