Multifeedback-Layer Neural Network

The architecture and training procedure of a novel recurrent neural network (RNN), referred to as the multifeedback-layer neural network (MFLNN), is described in this paper. The main difference of the proposed network compared to the available RNNs is that the temporal relations are provided by means of neurons arranged in three feedback layers, not by simple feedback elements, in order to enrich the representation capabilities of the recurrent networks. The feedback layers provide local and global recurrences via nonlinear processing elements. In these feedback layers, weighted sums of the delayed outputs of the hidden and of the output layers are passed through certain activation functions and applied to the feedforward neurons via adjustable weights. Both online and offline training procedures based on the backpropagation through time (BPTT) algorithm are developed. The adjoint model of the MFLNN is built to compute the derivatives with respect to the MFLNN weights which are then used in the training procedures. The Levenberg-Marquardt (LM) method with a trust region approach is used to update the MFLNN weights. The performance of the MFLNN is demonstrated by applying to several illustrative temporal problems including chaotic time series prediction and nonlinear dynamic system identification, and it performed better than several networks available in the literature

[1]  Eduardo D. Sontag,et al.  Neural Networks for Control , 1993 .

[2]  P. S. Sastry,et al.  Memory neuron networks for identification and control of dynamical systems , 1994, IEEE Trans. Neural Networks.

[3]  Ching-Hung Lee,et al.  Identification and control of dynamic systems using recurrent fuzzy neural networks , 2000, IEEE Trans. Fuzzy Syst..

[4]  Alberto Del Bimbo,et al.  Block-structured recurrent neural networks , 1995, Neural Networks.

[5]  Shun-Feng Su,et al.  On the dynamical modeling with neural fuzzy networks , 2002, IEEE Trans. Neural Networks.

[6]  N. K. Roy,et al.  Polymer property prediction and optimization using neural networks , 2006, IEEE Trans. Neural Networks.

[7]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[8]  Han-Xiong Li,et al.  An approximate internal model-based neural control for unknown nonlinear discrete processes , 2006, IEEE Transactions on Neural Networks.

[9]  Ah Chung Tsoi,et al.  Locally recurrent globally feedforward networks: a critical review of architectures , 1994, IEEE Trans. Neural Networks.

[10]  Giovanni Soda,et al.  Local Feedback Multilayered Networks , 1992, Neural Computation.

[11]  Jyh-Shing Roger Jang,et al.  ANFIS: adaptive-network-based fuzzy inference system , 1993, IEEE Trans. Syst. Man Cybern..

[12]  Jang-Hyun Park,et al.  Direct adaptive controller for nonaffine nonlinear systems using self-structuring neural networks , 2005, IEEE Transactions on Neural Networks.

[13]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[14]  Dan Wang,et al.  Neural network-based adaptive dynamic surface control for a class of uncertain nonlinear systems in strict-feedback form , 2005, IEEE Transactions on Neural Networks.

[15]  Albert Y. Zomaya,et al.  Adaptive Model-Based Control using Neural Networks , 1993 .

[16]  Michael I. Jordan.  Supervised learning and systems with excess degrees of freedom , 1988 .

[17]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[18]  Richard D. Braatz,et al.  On the "Identification and control of dynamical systems using neural networks" , 1997, IEEE Trans. Neural Networks.

[19]  George C. Mouzouris,et al.  Dynamic non-Singleton fuzzy logic systems for nonlinear modeling , 1997, IEEE Trans. Fuzzy Syst..

[20]  V I George,et al.  Introduction To Non–linear Optimization , 2009 .

[21]  Les E. Atlas,et al.  Recurrent neural networks and robust time series prediction , 1994, IEEE Trans. Neural Networks.

[22]  Chia-Feng Juang,et al.  A TSK-type recurrent fuzzy network for dynamic systems processing by neural network and genetic algorithms , 2002, IEEE Trans. Fuzzy Syst..

[23]  Jie Zhang,et al.  Recurrent neuro-fuzzy networks for nonlinear process modeling , 1999, IEEE Trans. Neural Networks.

[24]  Paris A. Mastorocostas,et al.  A recurrent fuzzy-neural model for dynamic system identification , 2002, IEEE Trans. Syst. Man Cybern. Part B.

[25]  Cheng-Jian Lin,et al.  Prediction and identification using wavelet-based recurrent fuzzy neural networks , 2004, IEEE Trans. Syst. Man Cybern. Part B.

[26]  Shuzhi Sam Ge,et al.  Adaptive neural control of uncertain MIMO nonlinear systems , 2004, IEEE Transactions on Neural Networks.

[27]  Peter Tino,et al.  IEEE Transactions on Neural Networks , 2009 .

[28]  Stephen A. Billings,et al.  Recurrent radial basis function networks for adaptive noise cancellation , 1995, Neural Networks.

[29]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[30]  Peter J. Gawthrop,et al.  Neural networks for control systems - A survey , 1992, Autom..

[31]  Chin-Teng Lin,et al.  A recurrent self-organizing neural fuzzy inference network , 1999, IEEE Trans. Neural Networks.

[32]  Bernard Widrow,et al.  Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights , 1990, 1990 IJCNN International Joint Conference on Neural Networks.