Analysis of Dynamical Recognizers

Pollack (1991) demonstrated that second-order recurrent neural networks can act as dynamical recognizers for formal languages when trained on positive and negative examples, and observed both phase transitions in learning and interacted function system-like fractal state sets. Follow on work focused mainly on the extraction and minimization of a finite state automaton (FSA) from the trained network. However, such networks are capable of inducing languages that are not regular and therefore not equivalent to any FSA. Indeed, it may be simpler for a small network to fit its training data by inducing such a nonregular language. But when is the network's language not regular? In this article, using a low-dimensional network capable of learning all the Tomita data sets, we present an empirical method for testing whether the language induced by the network is regular. We also provide a detailed "-machine analysis of trained networks for both regular and nonregular languages.

[1]  R. J. Nelson,et al.  Introduction to Automata , 1968 .

[2]  Boris A. Trakhtenbrot,et al.  Finite automata : behavior and synthesis , 1973 .

[3]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[4]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[5]  Michael F. Barnsley,et al.  Fractals everywhere , 1988 .

[6]  James L. McClelland,et al.  Finite State Automata and Simple Recurrent Networks , 1989, Neural Computation.

[7]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[8]  S. Fahlman.  Fast-learning variations on back propagation: an empirical study. , 1989 .

[9]  Michael I. Jordan.  Attractor dynamics and parallelism in a connectionist sequential machine , 1990 .

[10]  James P. Crutchfield,et al.  Computation at the Onset of Chaos , 1991 .

[11]  Hava T. Siegelmann,et al.  On the computational power of neural nets , 1992, COLT '92.

[12]  Kevin J. Lang.  Random DFA's can be approximately learned from sparse uniform examples , 1992, COLT '92.

[13]  C. Lee Giles,et al.  Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks , 1992, Neural Computation.

[14]  Raymond L. Watrous,et al.  Induction of Finite-State Languages Using Second-Order Recurrent Networks , 1992, Neural Computation.

[15]  Padhraic Smyth,et al.  Learning Finite State Machines With Self-Clustering Recurrent Networks , 1993, Neural Computation.

[16]  Michael C. Mozer,et al.  A Unified Gradient-Descent/Clustering Architecture for Finite State Machine Induction , 1993, NIPS.

[17]  John F. Kolen,et al.  Fool's Gold: Extracting Finite State Machines from Recurrent Network Dynamics , 1993, NIPS.

[18]  J. Crutchfield.  The calculi of emergence: computation, dynamics and induction , 1994 .

[19]  Panagiotis Manolios,et al.  First-Order Recurrent Neural Networks and Deterministic Finite State Automata , 1994, Neural Computation.

[20]  John F. Kolen,et al.  Exploring the computational capabilities of recurrent neural networks , 1995 .

[21]  Peter Tiňo,et al.  Finite State Machines and Recurrent Neural Networks -- Automata and Dynamical Systems Approaches , 1995 .

[22]  P. Tio,et al.  Learning and Extracting Initial Mealy Automata with a Modular Neural Network Model , 1995, Neural Computation.

[23]  Eduardo Sontag,et al.  Computational power of neural networks , 1995 .

[24]  Hava T. Siegelmann,et al.  On the Computational Power of Neural Nets , 1995, J. Comput. Syst. Sci..

[25]  Mikel L. Forcada,et al.  Learning the Initial State of a Second-Order Recurrent Neural Network during Regular-Language Inference , 1995, Neural Computation.

[26]  Giovanni Soda,et al.  Recurrent neural networks and prior knowledge for sequence processing: a constrained nondeterministic approach , 1995, Knowl. Based Syst..

[27]  Mike Casey,et al.  The Dynamics of Discrete-Time Computation, with Application to Recurrent Neural Networks and Finite State Machine Extraction , 1996, Neural Computation.

[28]  C. Lee Giles,et al.  Extraction of rules from discrete-time recurrent neural networks , 1996, Neural Networks.

[29]  Matthew Simon.  Finite State Automata , 1999 .