Identification using ANFIS with intelligent hybrid stable learning algorithm approaches and stability analysis of training methods

This paper proposes a novel hybrid learning algorithm with stable learning laws for Adaptive Network based Fuzzy Inference System (ANFIS) as a system identifier and studies the stability of this algorithm. The new hybrid learning algorithm is based on particle swarm optimization (PSO) for training the antecedent part and forgetting factor recursive least square (FFRLS) for training the conclusion part. Two famous training algorithms for ANFIS are the gradient descent (GD) to update antecedent part parameters and using GD or recursive least square (RLS) to update conclusion part parameters. Lyapunov stability theory is used to study the stability of the proposed algorithms. This paper, also studies the stability of PSO as an optimizer in training the identifier. Stable learning algorithms for the antecedent and consequent parts of fuzzy rules are proposed. Some constraints are obtained and simulation results are given to validate the results. It is shown that instability will not occur for the leaning rate and PSO factors in the presence of constraints. The learning rate can be calculated on-line and will provide an adaptive learning rate for the ANFIS structure. This new learning scheme employs adaptive learning rate that is determined by input-output data. Also, stable learning algorithms for two common methods are proposed based on Lyapunov stability theory and some constraints are obtained.

[1]  Marios M. Polycarpou,et al.  Learning and convergence analysis of neural-type structured networks , 1992, IEEE Trans. Neural Networks.

[2]  José Neves,et al.  The fully informed particle swarm: simpler, maybe better , 2004, IEEE Transactions on Evolutionary Computation.

[3]  Jie Zhang,et al.  Recurrent neuro-fuzzy networks for nonlinear process modeling , 1999, IEEE Trans. Neural Networks.

[4]  P. J. Angeline,et al.  Using selection to improve particle swarm optimization , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[5]  Jacek M. Zurada,et al.  An approach to multimodal biomedical image registration utilizing particle swarm optimization , 2004, IEEE Transactions on Evolutionary Computation.

[6]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[7]  Ronald R. Yager,et al.  Fuzzy sets, neural networks, and soft computing , 1994 .

[8]  J.-S.R. Jang,et al.  Input selection for ANFIS learning , 1996, Proceedings of IEEE 5th International Fuzzy Systems.

[9]  Zhongpeng Yang,et al.  A note on the trace inequality for products of Hermitian matrix power. , 2002 .

[10]  Ching-Hung Lee,et al.  Identification and control of dynamic systems using recurrent fuzzy neural networks , 2000, IEEE Trans. Fuzzy Syst..

[11]  Cheng-Jian Lin,et al.  Prediction and identification using wavelet-based recurrent fuzzy neural networks , 2004, IEEE Trans. Syst. Man Cybern. Part B.

[12]  Jujang Lee,et al.  Adaptive network-based fuzzy inference system with pruning , 2003, SICE 2003 Annual Conference (IEEE Cat. No.03TH8734).

[13]  Russell C. Eberhart,et al.  Multiobjective optimization using dynamic neighborhood particle swarm optimization , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[14]  Derek A. Linkens,et al.  Adaptive Weighted Particle Swarm Optimisation for Multi-objective Optimal Design of Alloy Steels , 2004, PPSN.

[15]  Vladimiro Miranda,et al.  EPSO-evolutionary particle swarm optimization, a new algorithm with applications in power systems , 2002, IEEE/PES Transmission and Distribution Conference and Exhibition.

[16]  Paris A. Mastorocostas,et al.  A recurrent fuzzy-neural model for dynamic system identification , 2002, IEEE Trans. Syst. Man Cybern. Part B.

[17]  E. Mizutani,et al.  Levenberg-Marquardt method for ANFIS learning , 1996, Proceedings of North American Fuzzy Information Processing.

[18]  T. R. Fortescue,et al.  Implementation of self-tuning regulators with variable forgetting factors , 1981, Autom..

[19]  L. Cao,et al.  A novel recursive algorithm for directional forgetting , 1999, Proceedings of the 1999 American Control Conference (Cat. No. 99CH36251).

[20]  Alberto Del Bimbo,et al.  Block-structured recurrent neural networks , 1995, Neural Networks.

[21]  James Kennedy,et al.  The Behavior of Particles , 1998, Evolutionary Programming.

[22]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[23]  R. Eberhart,et al.  Empirical study of particle swarm optimization , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[24]  Wook Hyun Kwon,et al.  Stability analysis and stabilization of fuzzy state space models , 1995 .

[25]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[26]  Yakov Frayman,et al.  A dynamically generated fuzzy neural network and its application to torsional vibration control of tandem cold rolling mill spindles , 2002 .

[27]  Mohammad Teshnehlab,et al.  Novel Hybrid Learning Algorithms for Tuning ANFIS Parameters Using Adaptive Weighted PSO , 2007, 2007 IEEE International Fuzzy Systems Conference.

[28]  R. Kulhavý.  Restricted exponential forgetting in real-time identification , 1985, at - Automatisierungstechnik.

[29]  Isao Hayashi,et al.  NN-driven fuzzy reasoning , 1991, Int. J. Approx. Reason..

[30]  H. Yoshida,et al.  A particle swarm optimization for reactive power and voltage control considering voltage security assessment , 1999, 2001 IEEE Power Engineering Society Winter Meeting. Conference Proceedings (Cat. No.01CH37194).

[31]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[32]  Shiyou Yang,et al.  A particle swarm optimization-based method for multiobjective design optimizations , 2005, IEEE Transactions on Magnetics.

[33]  Akio Nagamatsu,et al.  IMPROVED LEAST SQUARE IDENTIFICATION OF SOME STRUCTURAL PARAMETERS WITH A RECPTANCE REANALYSIS TECHNIQUE. , 1986 .

[34]  Gabriela Ciuprina,et al.  Use of intelligent-particle swarm optimization in electromagnetics. IEEE Trans Mag , 2002 .

[35]  Ching-Chih Tsai,et al.  Generalized predictive control using recurrent fuzzy neural networks for industrial processes , 2007 .

[36]  Saman K. Halgamuge,et al.  Self-organizing hierarchical particle swarm optimizer with time-varying acceleration coefficients , 2004, IEEE Transactions on Evolutionary Computation.

[37]  Anna G. Stefanopoulou,et al.  Recursive least squares with forgetting for online estimation of vehicle mass and road grade: theory and experiments , 2005 .

[38]  Jyh-Shing Roger Jang,et al.  ANFIS: adaptive-network-based fuzzy inference system , 1993, IEEE Trans. Syst. Man Cybern..

[39]  G. Goodwin,et al.  Modified least squares algorithm incorporating exponential resetting and forgetting , 1988 .

[40]  Russell C. Eberhart,et al.  Particle swarm with extended memory for multiobjective optimization , 2003, Proceedings of the 2003 IEEE Swarm Intelligence Symposium. SIS'03 (Cat. No.03EX706).

[41]  D. G. Fisher,et al.  Improved least squares identification , 1987 .

[42]  Yunfei Zhou,et al.  A new fuzzy neural network with fast learning algorithm and guaranteed stability for manufacturing process control , 2002, Fuzzy Sets Syst..

[43]  R. Eberhart,et al.  Comparing inertia weights and constriction factors in particle swarm optimization , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[44]  E. Ozcan,et al.  Particle swarm optimization: surfing the waves , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[45]  Yoshikazu Fukuyama,et al.  A Hybrid Particle Swarm Optimization for Distribution State Estimation , 2002, IEEE Power Engineering Review.

[46]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[47]  M. N. Vrahatis,et al.  Particle swarm optimization method in multiobjective problems , 2002, SAC '02.

[48]  M. Clerc,et al.  The swarm and the queen: towards a deterministic and adaptive particle swarm optimization , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[49]  Russell C. Eberhart,et al.  Human tremor analysis using particle swarm optimization , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[50]  Paolo Bolzern,et al.  Convergence and exponential convergence of identification algorithms with directional forgetting factor , 1990, Autom..

[51]  A. Sedigh,et al.  A novel training algorithm in ANFIS structure , 2006, 2006 American Control Conference.

[52]  Suganthan.  [IEEE 1999. Congress on Evolutionary Computation-CEC99 - Washington, DC, USA (6-9 July 1999)] Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406) - Particle swarm optimiser with neighbourhood operator , 1999 .

[53]  Carlos A. Coello Coello,et al.  Handling multiple objectives with particle swarm optimization , 2004, IEEE Transactions on Evolutionary Computation.

[54]  Kumpati S. Narendra,et al.  Neural Networks In Dynamical Systems , 1990, Other Conferences.

[55]  Chia-Feng Juang,et al.  Recurrent fuzzy network design using hybrid evolutionary learning algorithms , 2007, Neurocomputing.

[56]  Barak A. Pearlmutter.  Gradient calculations for dynamic recurrent neural networks: a survey , 1995, IEEE Trans. Neural Networks.

[57]  R. Kulhavý,et al.  On a general concept of forgetting , 1993 .

[58]  Xiaoou Li,et al.  Fuzzy identification using fuzzy neural networks with stable learning algorithms , 2004, IEEE Transactions on Fuzzy Systems.

[59]  G. M. Varazi,et al.  Constructive algorithm for neuro-fuzzy networks , 1997, Proceedings of 6th International Fuzzy Systems Conference.

[60]  Wen Yu,et al.  Fuzzy neural modeling using stable learning algorithm , 2003, Proceedings of the 2003 American Control Conference, 2003..

[61]  Ilya V. Kolmanovsky,et al.  Predictive energy management of a power-split hybrid electric vehicle , 2009, 2009 American Control Conference.

[62]  Vladimiro Miranda,et al.  EPSO - best-of-two-worlds meta-heuristic applied to power system problems , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[63]  Russell C. Eberhart,et al.  A new optimizer using particle swarm theory , 1995, MHS'95. Proceedings of the Sixth International Symposium on Micro Machine and Human Science.

[64]  Chia-Feng Juang,et al.  A recurrent self-organizing neural fuzzy inference network , 1997, Proceedings of 6th International Fuzzy Systems Conference.

[65]  Chia-Feng Juang,et al.  A TSK-type recurrent fuzzy network for dynamic systems processing by neural network and genetic algorithms , 2002, IEEE Trans. Fuzzy Syst..

[66]  T. Hägglund.  Recursive Estimation of Slowly Time-Varying Parameters , 1985 .

[67]  R. Garduno-Ramirez,et al.  Multiobjective control of power plants using particle swarm optimization techniques , 2006, IEEE Transactions on Energy Conversion.

[68]  Mu-Song Chen,et al.  A comparative study of learning methods in tuning parameters of fuzzy membership functions , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[69]  Visakan Kadirkamanathan,et al.  Stability analysis of the particle dynamics in particle swarm optimizer , 2006, IEEE Transactions on Evolutionary Computation.

[70]  A.P. Engelbrecht,et al.  Learning to play games using a PSO-based competitive learning approach , 2004, IEEE Transactions on Evolutionary Computation.

[71]  Keigo Watanabe,et al.  An optimized Takagi-Sugeno type neuro-fuzzy system for modeling robot manipulators , 2006, Neural Computing & Applications.