Training Recurrent Neural Networks Using Optimization Layer-by- Layer Recursive Least Squares Algorithm for Vibration Signals System Identification and Fault Diagnostic Analysis

Depending on the representation ability and the dynamic capability of recurrent neural networks for nonlinear models, a neural-based approach for general practical nonlinear systems identification is demonstrated in this paper. The paper presents a new method for the training of recurrent neural networks, namely the Optimization Layer-by-Layer Recursive Least Squares (OLL-RLS) algorithm. The method is based upon the weights to be optimized in layer-by-layer fashion, using the standard RLS method to speed up the convergence rate. The derivations of the OLL-RLS algorithm for the recurrent networks and the difficulties by the numerical instability and convergence stalling are also included and addressed in this paper. These difficulties can be tackled by introducing noise to the OLL-RLS algorithm, which enhances the convergence property significantly. Further, the convergence analysis and the computational complexity of the OLL-RLS algorithm are discussed. The validation results show that when applying to the systems identification, the novel OLL-RLS algorithm significantly outperforms other training algorithms, in both convergence speed and generalization capability. The proposed neural-based approach is also applied to the railway carriage system identification by identifying the train vibration signal. The identification results are very promising and show that the OLL-RLS algorithm can provide ^Corresponding author: e-mail: eetchow@cityu.edu.hk fax:: +852-27887791

[1]  Mahmood R. Azimi-Sadjadi,et al.  Fast learning process of multilayer neural networks using recursive least squares method , 1992, IEEE Trans. Signal Process..

[2]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[3]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[4]  Frank Bärmann,et al.  A learning algorithm for multilayered neural networks based on linear least squares problems , 1993, Neural Networks.

[5]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[6]  Gwilym M. Jenkins,et al.  Time series analysis, forecasting and control , 1971 .

[7]  Tommy W. S. Chow,et al.  Extended backpropagation algorithm , 1993 .

[8]  Simon Haykin,et al.  Adaptive filter theory (2nd ed.) , 1991 .

[9]  Tom Tollenaere,et al.  SuperSAB: Fast adaptive back propagation with good scaling properties , 1990, Neural Networks.

[10]  U. R. Prasad,et al.  Back propagation through adjoints for the identification of nonlinear dynamic systems using recurrent neural models , 1994, IEEE Trans. Neural Networks.

[11]  Anuradha M. Annaswamy,et al.  Stable Adaptive Systems , 1989 .

[12]  P. J. Dalianis,et al.  Fault diagnosis in complex systems using artificial neural networks , 1994, 1994 Proceedings of IEEE International Conference on Control and Applications.

[13]  Lee A. Feldkamp,et al.  Recurrent network training with the decoupled-extended-Kalman-filter algorithm , 1992, Defense, Security, and Sensing.

[14]  P. Vega,et al.  A neural networks based approach for fault detection and diagnosis: application to a real process , 1995, Proceedings of International Conference on Control Applications.

[15]  Rolf Isermann,et al.  Process fault detection based on modeling and estimation methods - A survey , 1984, Autom..

[16]  Luís B. Almeida,et al.  A learning rule for asynchronous perceptrons with feedback in a combinatorial environment , 1990 .

[17]  Tommy W. S. Chow,et al.  Extended least squares based algorithm for training feedforward networks , 1997, IEEE Trans. Neural Networks.

[18]  Tommy W. S. Chow,et al.  Neural network based short-term load forecasting using weather compensation , 1996 .

[19]  Paul M. Frank,et al.  Fault diagnosis in dynamic systems: theory and application , 1989 .

[20]  Lee A. Feldkamp,et al.  Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks , 1994, IEEE Trans. Neural Networks.

[21]  Fernando J. Pineda,et al.  GENERALIZATION OF BACKPROPAGATION TO RECURRENT AND HIGH-ORDER NETWORKS. , 1987 .

[22]  T. Chow,et al.  Nonlinear autoregressive integrated neural network model for short-term load forecasting , 1996 .

[23]  Pineda,et al.  Generalization of back-propagation to recurrent neural networks. , 1987, Physical review letters.

[24]  Timo Sorsa,et al.  Neural networks in process fault diagnosis , 1991, IEEE Trans. Syst. Man Cybern..

[25]  Michael Towsey,et al.  Training a neural network with conjugate gradient methods , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[26]  Rolf Isermann,et al.  Process Fault Detection Based on Modeling and Estimation Methods , 1982 .

[27]  Farmer,et al.  Predicting chaotic time series. , 1987, Physical review letters.

[28]  Gou Fei,et al.  On the identification of non-minimum phase non-gaussian MA and ARM A models using a third-order cumulant , 1995 .

[29]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[30]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[31]  A. Lapedes,et al.  Nonlinear Signal Processing Using Neural Networks , 1987 .

[32]  S. Ergezinger,et al.  An accelerated learning algorithm for multilayer perceptrons: optimization layer by layer , 1995, IEEE Trans. Neural Networks.

[33]  Kumpati S. Narendra,et al.  Identification Using Feedforward Networks , 1995, Neural Computation.