Constructing deterministic finite-state automata in recurrent neural networks

Recurrent neural networks that are <italic>trained</italic> to behave like deterministic finite-state automata (DFAs) can show deteriorating performance when tested on long strings. This deteriorating performance can be attributed to the instability of the internal representation of the learned DFA states. The use of a sigmoidel discriminant function together with the recurrent structure contribute to this instability. We prove that a simple algorithm can <italic>construct</italic> second-order recurrent neural networks with a sparse interconnection topology and sigmoidal discriminant function such that the internal DFA state representations are stable, that is, the constructed network correctly classifies strings of <italic>arbitrary length</italic>. The algorithm is based on encoding strengths of weights directly into the neural network. We derive a relationship between the weight strength and the number of DFA states for robust string classification. For a DFA with <italic>n</italic> state and <italic>m</italic>input alphabet symbols, the constructive algorithm generates a “programmed” neural network with <italic>O</italic>(<italic>n</italic>) neurons and <italic>O</italic>(<italic>mn</italic>) weights. We compare our algorithm to other methods proposed in the literature.

[1]  Raymond L. Watrous,et al.  Induction of Finite-State Languages Using Second-Order Recurrent Networks , 1992, Neural Computation.

[2]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[3]  Bing J. Sheu,et al.  Neural information processing and VLSI , 1995 .

[4]  Jude Shavlik,et al.  Refinement ofApproximate Domain Theories by Knowledge-Based Neural Networks , 1990, AAAI.

[5]  Peter Tiňo,et al.  Fixed points in two-neuron discrete time recurrent networks: stability and bifurcation considerations , 1995 .

[6]  Morris W. Hirsch,et al.  Saturation at high gain in discrete time recurrent networks , 1994, Neural Networks.

[7]  Giovanni Soda,et al.  An unified approach for integrating explicit knowledge and learning by example in recurrent networks , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[8]  C. L. Giles,et al.  Second-order recurrent neural networks for grammatical inference , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[9]  C. Lee Giles,et al.  Rule Revision With Recurrent Neural Networks , 1996, IEEE Trans. Knowl. Data Eng..

[10]  Michael F. Barnsley,et al.  Fractals everywhere , 1988 .

[11]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[12]  C. L. Giles,et al.  Dynamic recurrent neural networks: Theory and applications , 1994, IEEE Trans. Neural Networks Learn. Syst..

[13]  C. Lee Giles,et al.  Stable Encoding of Large Finite-State Automata in Recurrent Neural Networks with Sigmoid Discriminants , 1996, Neural Computation.

[14]  C. L. Giles,et al.  Rule refinement with recurrent neural networks , 1993, IEEE International Conference on Neural Networks.

[15]  Marvin Minsky,et al.  Computation : finite and infinite machines , 2016 .

[16]  C. L. Giles,et al.  Inserting rules into recurrent neural networks , 1992, Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop.

[17]  Don R. Hush,et al.  Bounds on the complexity of recurrent neural network implementations of finite state machines , 1993, Neural Networks.

[18]  M. W. Shields.  An Introduction to Automata Theory , 1988 .

[19]  Padhraic Smyth,et al.  Learning Finite State Machines With Self-Clustering Recurrent Networks , 1993, Neural Computation.

[20]  Morris W. Hirsch,et al.  Convergent activation dynamics in continuous time networks , 1989, Neural Networks.

[21]  Mike Casey,et al.  The Dynamics of Discrete-Time Computation, with Application to Recurrent Neural Networks and Finite State Machine Extraction , 1996, Neural Computation.

[22]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[23]  Carver Mead,et al.  Analog VLSI and neural systems , 1989 .

[24]  Richard Maclin,et al.  Refining algorithms with knowledge-based neural networks: improving the Chou-Fasman algorithm for protein folding , 1994, COLT 1994.

[25]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[26]  Noga Alon,et al.  Efficient simulation of finite automata by neural nets , 1991, JACM.

[27]  C. Lee Giles,et al.  Training Second-Order Recurrent Neural Networks using Hints , 1992, ML.

[28]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[29]  C. Lee Giles,et al.  Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks , 1992, Neural Computation.