What NARX Networks Can Compute

We prove that a class of architectures called NARX neural networks, popular in control applications and other problems, are at least as powerful as fully connected recurrent neural networks. Recent results have shown that fully connected networks are Turing equivalent. Building on those results, we prove that NARX networks are also universal computation devices. NARX networks have a limited feedback which comes only from the output neuron rather than from hidden states. There is much interest in the amount and type of recurrence to be used in recurrent neural networks. Our results pose the question of what amount of feedback or recurrence is necessary for any network to be Turing equivalent and what restrictions on feedback limit computational power.

[1]  Don R. Hush,et al.  Bounds on the complexity of recurrent neural network implementations of finite state machines , 1993, Neural Networks.

[2]  Hava T. Siegelmann,et al.  Analog computation via neural networks , 1993, [1993] The 2nd Israel Symposium on Theory and Computing Systems.

[3]  Stephen A. Billings,et al.  Non-linear system identification using neural networks , 1990 .

[4]  Marvin Minsky,et al.  Computation : finite and infinite machines , 2016 .

[5]  Noga Alon,et al.  Efficient simulation of finite automata by neural nets , 1991, JACM.

[6]  Giovanni Soda,et al.  Local Feedback Multilayered Networks , 1992, Neural Computation.

[7]  C. Lee Giles,et al.  An experimental comparison of recurrent neural networks , 1994, NIPS.

[8]  Hava T. Siegelmann,et al.  On the Computational Power of Neural Nets , 1995, J. Comput. Syst. Sci..

[9]  Les E. Atlas,et al.  Recurrent Networks and NARMA Modeling , 1991, NIPS.

[10]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[11]  S Z Qin,et al.  Comparison of four neural net learning methods for dynamic system identification , 1992, IEEE Trans. Neural Networks.

[12]  Hava T. Siegelmann,et al.  On the power of sigmoid neural networks , 1993, COLT '93.

[13]  P. Werbos,et al.  Long-term predictions of chemical processes using recurrent neural networks: a parallel training approach , 1992 .

[14]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[15]  I. J. Leontaritis,et al.  Input-output parametric models for non-linear systems Part II: stochastic non-linear systems , 1985 .

[16]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[17]  Hava T. Siegelmann,et al.  The complexity of language recognition by neural networks , 1992, Neurocomputing.

[18]  José Carlos Príncipe,et al.  The gamma model--A new neural model for temporal processing , 1992, Neural Networks.

[19]  Ah Chung Tsoi,et al.  FIR and IIR Synapses, a New Neural Network Architecture for Time Series Modeling , 1991, Neural Computation.

[20]  C. Lee Giles,et al.  Stable Encoding of Large Finite-State Automata in Recurrent Neural Networks with Sigmoid Discriminants , 1996, Neural Computation.

[21]  C. Lee Giles,et al.  Learning a class of large finite state machines with a recurrent neural network , 1995, Neural Networks.