Robust adaptive learning of feedforward neural networks via LMI optimizations

Feedforward neural networks (FNNs) have been extensively applied to various areas such as control, system identification, function approximation, pattern recognition etc. A novel robust control approach to the learning problems of FNNs is further investigated in this study in order to develop efficient learning algorithms which can be implemented with optimal parameter settings and considering noise effect in the data. To this aim, the learning problem of a FNN is cast into a robust output feedback control problem of a discrete time-varying linear dynamic system. New robust learning algorithms with adaptive learning rate are therefore developed, using linear matrix inequality (LMI) techniques to find the appropriate learning rates and to guarantee the fast and robust convergence. Theoretical analysis and examples are given to illustrate the theoretical results.

[1]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[2]  Stefen Hui,et al.  Application of feedforward neural networks to dynamical system identification and control , 1993, IEEE Trans. Control. Syst. Technol..

[3]  Miguel Pinzolas,et al.  Neighborhood based Levenberg-Marquardt algorithm for neural network training , 2002, IEEE Trans. Neural Networks.

[4]  Kiyoshi Nishiyama,et al.  H∞-learning of layered neural networks , 2001, IEEE Trans. Neural Networks.

[5]  Laxmidhar Behera,et al.  On Adaptive Learning Rate That Guarantees Convergence in Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[6]  R. Sutton.  Gain Adaptation Beats Least Squares , 2006 .

[7]  C. Abdallah,et al.  Static Output Feedback : A Survey 1 , 2009 .

[8]  P. S. Sastry,et al.  Analysis of the back-propagation algorithm with momentum , 1994, IEEE Trans. Neural Networks.

[9]  Peter Sarnak,et al.  Linking Numbers of Modular Knots , 2010 .

[10]  O. Nelles.  Nonlinear System Identification , 2001 .

[11]  Xingjian Jing.  An H∞ control approach to robust learning of feedforward neural networks , 2011, Neural Networks.

[12]  Hideaki Sakai,et al.  A real-time learning algorithm for a multilayered neural network based on the extended Kalman filter , 1992, IEEE Trans. Signal Process..

[13]  Stanislaw Osowski,et al.  Fast Second Order Learning Algorithm for Feedforward Multilayer Neural Networks and its Applications , 1996, Neural Networks.

[14]  Martin T. Hagan,et al.  Neural network design , 1995 .

[15]  Francisco López-Ferreras,et al.  Study of Two Error Functions to Approximate the Neyman–Pearson Detector Using Supervised Learning Machines , 2009, IEEE Transactions on Signal Processing.

[16]  P. Shi,et al.  Guaranteed cost control of uncertain discrete-time systems with delay in state , 1999 .

[17]  Stephen A. Billings,et al.  Model structure selection using an integrated forward orthogonal search algorithm assisted by squared correlation and mutual information , 2008, Int. J. Model. Identif. Control..

[18]  Dilip Sarkar,et al.  Methods to speed up error back-propagation learning algorithm , 1995, CSUR.

[19]  Hugang Han,et al.  Adaptive control of a class of nonlinear systems with nonlinearly parameterized fuzzy approximators , 2001, IEEE Trans. Fuzzy Syst..

[20]  E. Yaz.  Linear Matrix Inequalities In System And Control Theory , 1998, Proceedings of the IEEE.

[21]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[22]  Wei Wu,et al.  Convergence of BP Algorithm with Variable Learning Rates for FNN Training , 2006, 2006 Fifth Mexican International Conference on Artificial Intelligence.

[23]  Quoc V. Le,et al.  Proximal regularization for online and batch learning , 2009, ICML '09.

[24]  Patrick Gallinari,et al.  SGD-QN: Careful Quasi-Newton Stochastic Gradient Descent , 2009, J. Mach. Learn. Res..

[25]  Zhihong Man,et al.  A New Adaptive Backpropagation Algorithm Based on Lyapunov Stability Theory for Neural Networks , 2006, IEEE Transactions on Neural Networks.

[26]  B. Subudhi,et al.  Estimation of power system harmonics using hybrid RLS-Adaline and KF-Adaline algorithms , 2009, TENCON 2009 - 2009 IEEE Region 10 Conference.

[27]  C. Charalambous,et al.  Conjugate gradient algorithm for efficient training of artifi-cial neural networks , 1990 .

[28]  Stephen A. Billings,et al.  An adaptive wavelet neural network for spatio-temporal system identification , 2010, Neural Networks.

[29]  Mark Kon,et al.  Regularization Techniques for Machine Learning on Graphs and Networks with Biological Applications , 2010 .

[30]  Sheng Chen,et al.  Sparse modeling using orthogonal forward regression with PRESS statistic and regularization , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[31]  Leszek Rutkowski,et al.  A fast training algorithm for neural networks , 1998 .

[32]  Stephen P. Boyd,et al.  Linear Matrix Inequalities in Systems and Control Theory , 1994 .