Learning, Representation, and Synthesis of Discrete Dynamical Systems in Continuous Recurrent Neural

| This paper gives an overview on learning and representation of discrete-time, discrete-space dynamical systems in discrete-time, continuous-space recurrent neural networks. We limit our discussion to dynami-cal systems (recurrent neural networks) which can be represented as nite-state machines (e.g. discrete event systems 53]). In particular , we discuss how a symbolic representation of the learned states and dynamics can be extracted from trained neural networks, and how (partially) known deterministic nite-state automata (DFAs) can be encoded in recurrent networks. While the DFAs that can be learned exactly with recurrent neural networks are generally small (on the order of 20 states), there exist subclasses of DFAs with on the order of 1000 states that can be learned by small recurrent networks. However, recent work in natural language processing implies that recurrent networks can possibly learn larger state systems 35].

[1]  Hava T. Siegelmann,et al.  Computational capabilities of recurrent NARX neural networks , 1997, IEEE Trans. Syst. Man Cybern. Part B.

[2]  Padhraic Smyth,et al.  Discrete recurrent neural networks for grammatical inference , 1994, IEEE Trans. Neural Networks.

[3]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[4]  Christof Koch,et al.  Seeing Chips: Analog VLSI Circuits for Computer Vision , 1989, Neural Computation.

[5]  Learning and Extracting Initial Mealy Automata with a Modular Neural Network Model , 1995, Neural Computation.

[6]  Padhraic Smyth,et al.  Learning Finite State Machines With Self-Clustering Recurrent Networks , 1993, Neural Computation.

[7]  P. Ashar,et al.  Sequential Logic Synthesis , 1991 .

[8]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[9]  W. Omlin.  Fault-tolerant Implementation of Finite-state Automata in Recurrent Neural Networks , 1995 .

[10]  Lawrence D. Jackel,et al.  VLSI implementation of a neural network model , 1988, Computer.

[11]  Pierre Roussel-Ragot,et al.  Neural Networks and Nonlinear Adaptive Filtering: Unifying Concepts and New Algorithms , 1993, Neural Computation.

[12]  C. Lee Giles,et al.  Learning a class of large finite state machines with a recurrent neural network , 1995, Neural Networks.

[13]  C. Lee Giles,et al.  Using recurrent neural networks to learn the structure of interconnection networks , 1995, Neural Networks.

[14]  James P. Crutchfield,et al.  Computation at the Onset of Chaos , 1991 .

[15]  C. L. Giles,et al.  Heuristics for the extraction of rules from discrete-time recurrent neural networks , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[16]  Yaser S. Abu-Mostafa,et al.  Learning from hints in neural networks , 1990, J. Complex..

[17]  Bing J. Sheu,et al.  Neural information processing and VLSI , 1995 .

[18]  Anil Nerode,et al.  Models for Hybrid Systems: Automata, Topologies, Controllability, Observability , 1992, Hybrid Systems.

[19]  Michael C. Mozer,et al.  A Unified Gradient-Descent/Clustering Architecture for Finite State Machine Induction , 1993, NIPS.

[20]  Raymond A. DeCarlo,et al.  Analysis of a hybrid system using symbolic dynamics and Petri Nets , 1994, Autom..

[21]  Petr Lisoněk,et al.  Symbolic Computation Approach to Nonlinear Dynamics , 1991 .

[22]  Michael H. Freedman,et al.  Computation in discrete-time dynamical systems , 1995 .

[23]  Richard Maclin,et al.  Refining algorithms with knowledge-based neural networks: improving the Chou-Fasman algorithm for protein folding , 1994, COLT 1994.

[24]  Yu-Chi Ho,et al.  Discrete event dynamic systems : analyzing complexity and performance in the modern world , 1992 .

[25]  C. L. Giles,et al.  Dynamic recurrent neural networks: Theory and applications , 1994, IEEE Trans. Neural Networks Learn. Syst..

[26]  Peter Tiňo,et al.  Finite State Machines and Recurrent Neural Networks -- Automata and Dynamical Systems Approaches , 1995 .

[27]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[28]  Carver Mead,et al.  Analog VLSI and neural systems , 1989 .

[29]  R. O. Grondin,et al.  VLSI Implementation of Neural Classifiers , 1990, Neural Computation.

[30]  C. Lee Giles,et al.  Stable Encoding of Large Finite-State Automata in Recurrent Neural Networks with Sigmoid Discriminants , 1996, Neural Computation.

[31]  Kenneth Man-kam Yip,et al.  Understanding Complex Dynamics by Visual and Symbolic Reasoning , 1991, Artif. Intell..

[32]  J. Kolen.  Recurrent Networks: State Machines Or Iterated Function Systems? , 1994 .

[33]  Ronald J. Williams,et al.  Gradient-based learning algorithms for recurrent networks and their computational complexity , 1995 .

[34]  C. Lee Giles,et al.  Extraction of rules from discrete-time recurrent neural networks , 1996, Neural Networks.

[35]  Raymond L. Watrous,et al.  Induction of Finite-State Languages Using Second-Order Recurrent Networks , 1992, Neural Computation.

[36]  Jana Kosecka,et al.  Control of Discrete Event Systems , 1992 .

[37]  C. Lee Giles,et al.  An experimental comparison of recurrent neural networks , 1994, NIPS.

[38]  A. Meystel,et al.  Multiscale models and controllers , 1994, Proceedings of IEEE Symposium on Computer-Aided Control Systems Design (CACSD).

[39]  Marwan A. Jabri,et al.  Weight perturbation: an optimal architecture and learning technique for analog VLSI feedforward and recurrent multilayer networks , 1992, IEEE Trans. Neural Networks.

[40]  Yannis P. Tsividis,et al.  A Reconfigurable Analog VLSI Neural Network Chip , 1989, NIPS.

[41]  C. L. Giles,et al.  Inserting rules into recurrent neural networks , 1992, Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop.

[42]  Dan Hammerstrom,et al.  Fault simulation of a wafer-scale integrated neural network , 1988, Neural Networks.

[43]  C. Lee Giles,et al.  Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks , 1992, Neural Computation.

[44]  C. Lee Giles,et al.  Higher Order Recurrent Networks and Grammatical Inference , 1989, NIPS.

[45]  Kenji Doya,et al.  Universality of Fully-Connected Recurrent Neural Networks , 1993 .

[46]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[47]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[48]  Hava T. Siegelmann,et al.  On the Computational Power of Neural Nets , 1995, J. Comput. Syst. Sci..