Bounds on the complexity of recurrent neural network implementations of finite state machines

Abstract In this paper the efficiency of recurrent neural network implementations of m-state finite state machines will be explored. Specifically, it will be shown that the node complexity for the unrestricted case can be bounded above by O (√ m ). It will also be shown that the node complexity is O( m log m ) when the weights and thresholds are restricted to the set {−1,1} and O(m) when the fan-in is restricted to two. Matching lower bounds will be provided for each of these upper bounds assuming that the state of the FSM can be encoded in a subset of the nodes of size [ log m].

[1]  Michael I. Jordan,et al.  Forward Models: Supervised Learning with a Distal Teacher , 1992, Cogn. Sci..

[2]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[3]  Claude E. Shannon,et al.  The synthesis of two-terminal switching circuits , 1949, Bell Syst. Tech. J..

[4]  Hava T. Siegelmann,et al.  On the power of sigmoid neural networks , 1993, COLT '93.

[5]  Hava T. Siegelmann,et al.  The complexity of language recognition by neural networks , 1992, Neurocomputing.

[6]  Richard D. Braatz,et al.  On the "Identification and control of dynamical systems using neural networks" , 1997, IEEE Trans. Neural Networks.

[7]  Don R. Hush,et al.  On the node complexity of neural networks , 1994, Neural Networks.

[8]  O. B. Lupanov,et al.  Circuits Using Threshold Elements. , 1972 .

[9]  C. Lee Giles,et al.  Higher Order Recurrent Networks and Grammatical Inference , 1989, NIPS.

[10]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[11]  Anthony J. Robinson,et al.  Static and Dynamic Error Propagation Networks with Application to Speech Coding , 1987, NIPS.

[12]  Robert O. Winder,et al.  Bounds on Threshold Gate Realizability , 1963, IEEE Transactions on Electronic Computers.

[13]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Saburo Muroga,et al.  Threshold logic and its applications , 1971 .

[15]  Ah Chung Tsoi,et al.  FIR and IIR Synapses, a New Neural Network Architecture for Time Series Modeling , 1991, Neural Computation.

[16]  Hava T. Siegelmann,et al.  Analog computation via neural networks , 1993, [1993] The 2nd Israel Symposium on Theory and Computing Systems.

[17]  Raymond L. Watrous,et al.  Induction of Finite-State Languages Using Second-Order Recurrent Networks , 1992, Neural Computation.

[18]  Pierre Roussel-Ragot,et al.  Neural Networks and Nonlinear Adaptive Filtering: Unifying Concepts and New Algorithms , 1993, Neural Computation.

[19]  Noga Alon,et al.  Efficient simulation of finite automata by neural nets , 1991, JACM.

[20]  Robert O. Winder,et al.  Threshold Logic Asymptotes , 1970, IEEE Transactions on Computers.

[21]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[22]  Eduardo Sontag,et al.  Computational power of neural networks , 1995 .

[23]  Ian Parberry,et al.  Circuit complexity and neural networks , 1994 .

[24]  Hava T. Siegelmann,et al.  On the Computational Power of Neural Nets , 1995, J. Comput. Syst. Sci..

[25]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.