Artificial neural network training utilizing the smooth variable structure filter estimation strategy

Abstract A multilayered neural network is a multi-input, multi-output nonlinear system in which network weights can be trained by using parameter estimation algorithms. In this paper, a novel training method is proposed. This method is based on the relatively new smooth variable structure filter (SVSF) and is formulated for feed-forward multilayer perceptron training. The SVSF is a state and parameter estimation that is based on the sliding mode concept and works in a predictor–corrector fashion. The SVSF training performance is tested on three benchmark pattern classification problems. Furthermore, a study is presented comparing the popular back-propagation method, the extended Kalman filter, and the SVSF.

[1]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[2]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[3]  B. Anderson,et al.  Optimal Filtering , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[4]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[5]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[6]  Raymond L. Watrous.  Learning Algorithms for Connectionist Networks: Applied Gradient Methods of Nonlinear Optimization , 1988 .

[7]  Sharad Singhal,et al.  Training Multilayer Perceptrons with the Extende Kalman Algorithm , 1988, NIPS.

[8]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[9]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[10]  Lee A. Feldkamp,et al.  Decoupled extended Kalman filter training of feedforward layered networks , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[11]  Ronald J. Williams,et al.  Training recurrent networks using the extended Kalman filter , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[12]  Hideaki Sakai,et al.  A real-time learning algorithm for a multilayered neural network based on the extended Kalman filter , 1992, IEEE Trans. Signal Process..

[13]  Michael Y. Hu,et al.  Two-Group Classification Using Neural Networks* , 1993 .

[14]  G. V. Puskorius,et al.  Training of robust neural controllers , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[15]  G. V. Puskorius,et al.  Training controllers for robustness: multi-stream DEKF , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[16]  Lutz Prechelt,et al.  A Set of Neural Network Benchmark Problems and Benchmarking Rules , 1994 .

[17]  Mohammad Bagher Menhaj,et al.  Training feedforward networks with the Marquardt algorithm , 1994, IEEE Trans. Neural Networks.

[18]  Martin T. Hagan,et al.  Neural network design , 1995 .

[19]  Rudy Setiono,et al.  Use of a quasi-Newton method in a feedforward neural network construction algorithm , 1995, IEEE Trans. Neural Networks.

[20]  H.F. Durrant-Whyte,et al.  A new approach for filtering nonlinear systems , 1995, Proceedings of 1995 American Control Conference - ACC'95.

[21]  Brad Warner,et al.  Understanding Neural Networks as Statistical Tools , 1996 .

[22]  Jennie Si,et al.  Advanced neural-network training algorithm with reduced complexity based on Jacobian deficiency , 1998, IEEE Trans. Neural Networks.

[23]  Dietmar Heinke,et al.  Comparing neural networks: a benchmark on growing neural gas, growing cell structures, and fuzzy ARTMAP , 1998, IEEE Trans. Neural Networks.

[24]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[25]  Felix Heimes,et al.  Extended Kalman filter neural network training: experimental results and algorithm improvements , 1998, SMC'98 Conference Proceedings. 1998 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.98CH36218).

[26]  G. V. Puskorius,et al.  A signal processing framework based on dynamic neural networks with application to problems in adaptation, filtering, and classification , 1998, Proc. IEEE.

[27]  Rudolph van der Merwe,et al.  The unscented Kalman filter for nonlinear estimation , 2000, Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No.00EX373).

[28]  Simon Haykin,et al.  A Signal Processing Framework Based on Dynamic Neural Networks with Application to Problems in Adaptation, Filtering and Classification , 2001 .

[29]  S. Haykin.  Kalman Filtering and Neural Networks , 2001 .

[30]  Danil V. Prokhorov,et al.  Neural network training with the nprKF , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[31]  Saeid Habibi,et al.  The Variable Structure Filter , 2002 .

[32]  Antônio de Pádua Braga,et al.  Control of a Robotic Manipulator Using Artificial Neural Networks with On-line Adaptation , 2000, Neural Processing Letters.

[33]  T.,et al.  Training Feedforward Networks with the Marquardt Algorithm , 2004 .

[34]  Xie Jian-ying,et al.  A new learning algorithm for diagonal recurrent neural network , 2005 .

[35]  Weizhong Guo,et al.  A New Learning Algorithm for Diagonal Recurrent Neural Network , 2005, ICNC.

[36]  Runxuan Zhang,et al.  Efficient sequential and batch learning artificial neural network methods for classification problems , 2006 .

[37]  Le Yang,et al.  Development of A New Recurrent Neural Network Toolbox (RNN-Tool) ⁄ , 2006 .

[38]  Shu Wang,et al.  A Smooth Variable Structure filter for State estimation , 2006, Control. Intell. Syst..

[39]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation (3rd Edition) , 2007 .

[40]  Saeid Habibi,et al.  The Smooth Variable Structure Filter , 2007, Proceedings of the IEEE.

[41]  Saeid Habibi,et al.  Parameter Identification for a High-Performance Hydrostatic Actuation System Using the Variable Structure Filter Concept , 2007 .

[42]  Jiri Pospichal,et al.  Neural Network Training with Extended Kalman Filter Using Graphics Processing Unit , 2008, ICANN.

[43]  S. Habibi,et al.  Target Tracking Using the Smooth Variable Structure Filter , 2009 .

[44]  S. Andrew Gadsden,et al.  A new form of the smooth variable structure filter with a covariance derivation , 2010, 49th IEEE Conference on Decision and Control (CDC).

[45]  S. Andrew Gadsden,et al.  Fault detection of an engine using a neural network trained by the smooth variable structure filter , 2011, 2011 IEEE International Conference on Control Applications (CCA).

[46]  Stephen Andrew Gadsden,et al.  SMOOTH VARIABLE STRUCTURE FILTERING: THEORY AND APPLICATIONS , 2011 .

[47]  Saeid Habibi,et al.  A New Robust Filtering Strategy for Linear Systems , 2012 .

[48]  Saeid R. Habibi,et al.  Novel Model-Based Estimators for the Purposes of Fault Detection and Diagnosis , 2013, IEEE/ASME Transactions on Mechatronics.

[49]  Jimi Tjong,et al.  Automotive Internal-Combustion-Engine Fault Detection and Classification Using Artificial Neural Network Techniques , 2015, IEEE Transactions on Vehicular Technology.