Link Between and Comparison and Combination of Zhang Neural Network and Quasi-Newton BFGS Method for Time-Varying Quadratic Minimization

Since 2001, a novel type of recurrent neural network called Zhang neural network (ZNN) has been proposed, investigated, and exploited for solving online time-varying problems in a variety of scientific and engineering fields. In this paper, three discrete-time ZNN models are first proposed to solve the problem of time-varying quadratic minimization (TVQM). Such discrete-time ZNN models exploit methodologically the time derivatives of time-varying coefficients and the inverse of the time-varying coefficient matrix. To eliminate explicit matrix-inversion operation, the quasi-Newton BFGS method is introduced, which approximates effectively the inverse of the Hessian matrix; thus, three discrete-time ZNN models combined with the quasi-Newton BFGS method (named ZNN-BFGS) are proposed and investigated for TVQM. In addition, according to the criterion of whether the time-derivative information of time-varying coefficients is explicitly known/used or not, these proposed discrete-time models are classified into three categories: 1) models with time-derivative information known (i.e., ZNN-K and ZNN-BFGS-K models), 2) models with time-derivative information unknown (i.e., ZNN-U and ZNN-BFGS-U models), and 3) simplified models without using time-derivative information (i.e., ZNN-S and ZNN-BFGS-S models). The well-known gradient-based neural network is also developed to handle TVQM for comparison with the proposed ZNN and ZNN-BFGS models. Illustrative examples are provided and analyzed to substantiate the efficacy of these proposed models for TVQM.

[1]  Dongsheng Guo,et al.  More than Newton iterations generalized from Zhang neural network for constant matrix inversion aided with line-search algorithm , 2010, IEEE ICCA 2010.

[2]  Yunong Zhang,et al.  O(N 2)-Operation Approximation of Covariance Matrix Inverse in Gaussian Process Regression Based on Quasi-Newton BFGS Method , 2007, Commun. Stat. Simul. Comput..

[3]  LI Hai-lin.  Verification and Practice on First-Order Numerical Differentiation Formulas for Unknown Target Functions , 2009 .

[4]  Rusli,et al.  Feedforward Neural Network Trained by BFGS Algorithm for Modeling Plasma Etching of Silicon Carbide , 2010, IEEE Transactions on Plasma Science.

[5]  Yunong Zhang,et al.  Zhang neural network for online solution of time-varying convex quadratic program subject to time-varying linear-equality constraints , 2009 .

[6]  Yunong Zhang,et al.  Convergence analysis of Zhang neural networks solving time-varying linear equations but without using time-derivative information , 2010, IEEE ICCA 2010.

[7]  Youshen Xia,et al.  An improved neural network for convex quadratic optimization with application to real-time beamforming , 2005, Neurocomputing.

[8]  Sun Yat-sen,et al.  Approximation-Performance and Global-Convergence Analysis of Basis-Function Feedforward Neural Network , 2009 .

[9]  Yunong Zhang,et al.  Time-series Gaussian Process Regression Based on Toeplitz Computation of O(N2) Operations and O(N)-level Storage , 2005, Proceedings of the 44th IEEE Conference on Decision and Control.

[10]  María José Pérez-Ilzarbe.  Convergence analysis of a discrete-time recurrent neural network to perform quadratic real optimization with bound constraints , 1998, IEEE Trans. Neural Networks.

[11]  D. Goldfarb.  A family of variable-metric methods derived by variational means , 1970 .

[12]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[13]  Peng Xu,et al.  Time-varying square roots finding via Zhang dynamics versus gradient dynamics and the former's link and new explanation to Newton-Raphson iteration , 2010, Inf. Process. Lett..

[14]  Shuzhi Sam Ge,et al.  Design and analysis of a general recurrent neural network model for time-varying matrix inversion , 2005, IEEE Transactions on Neural Networks.

[15]  Shuzhi Sam Ge,et al.  A unified quadratic-programming-based dynamical system approach to joint torque optimization of physically constrained redundant manipulators , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[16]  Yu-Nong Zhang,et al.  Zhang Neural Network for Linear Time-Varying Equation Solving and its Robotic Application , 2007, 2007 International Conference on Machine Learning and Cybernetics.

[17]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 2. The New Algorithm , 1970 .

[18]  Peter C. Jurs,et al.  Simulation of polysaccharide carbon-13 nuclear magnetic resonance spectra using regression analysis and neural networks , 1993 .

[19]  Binghuang Cai,et al.  Common nature of learning between BP and hopfield-type neural networks for convex quadratic minimization with simplified network models , 2009, 2009 International Conference on Mechatronics and Automation.

[20]  Rudy Setiono,et al.  Use of a quasi-Newton method in a feedforward neural network construction algorithm , 1995, IEEE Trans. Neural Networks.

[21]  Yunong Zhang,et al.  Improved Zhang neural network model and its solution of time-varying generalized linear matrix equations , 2010, Expert Syst. Appl..

[22]  D. Shanno.  Conditioning of Quasi-Newton Methods for Function Minimization , 1970 .

[23]  Ke Chen,et al.  Zhang Neural Network Versus Gradient Neural Network for Online Time-Varying Quadratic Function Minimization , 2008, ICIC.

[24]  Zhipeng Wu,et al.  BFGS quasi-Newton method for solving electromagnetic inverse problems , 2006 .

[25]  Dongsheng Guo,et al.  Comparison on Continuous-Time Zhang Dynamics and Newton-Raphson Iteration for Online Solution of Nonlinear Equations , 2011, ISNN.

[26]  Xiaolin Hu,et al.  An Alternative Recurrent Neural Network for Solving Variational Inequalities and Related Optimization Problems , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[27]  Carver Mead,et al.  Analog VLSI and neural systems , 1989 .

[28]  Michael A. Arbib,et al.  The handbook of brain theory and neural networks , 1995, A Bradford book.

[29]  Xue-Bin Liang,et al.  Improved upper bound on step-size parameters of discrete-time recurrent neural networks for linear inequality and equation system , 2002 .

[30]  Xiaolin Hu,et al.  Design of General Projection Neural Networks for Solving Monotone Linear Variational Inequalities and Linear and Quadratic Optimization Problems , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[31]  R. Fletcher,et al.  A New Approach to Variable Metric Algorithms , 1970, Comput. J..

[32]  Ke Chen,et al.  Performance Analysis of Gradient Neural Network Exploited for Online Time-Varying Matrix Inversion , 2009, IEEE Transactions on Automatic Control.

[33]  Yunong Zhang,et al.  Robustness analysis of the Zhang neural network for online time-varying quadratic optimization , 2010 .

[34]  Jun Wang,et al.  A one-layer recurrent neural network for support vector machine learning , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[35]  Abhishek K Gupta,et al.  Numerical Methods using MATLAB , 2014, Apress.

[36]  Frank L. Lewis,et al.  Optimal design of CMAC neural-network controller for robot manipulators , 2000, IEEE Trans. Syst. Man Cybern. Part C.

[37]  Yunong Zhang.  Towards Piecewise-Linear Primal Neural Networks for Optimization and Redundant Robotics , 2006, 2006 IEEE International Conference on Networking, Sensing and Control.

[38]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations , 1970 .

[39]  Binghuang Cai,et al.  From Zhang Neural Network to Newton Iteration for Matrix Inversion , 2009, IEEE Transactions on Circuits and Systems I: Regular Papers.