A Gradient-Based Neural Network Method for Solving Strictly Convex Quadratic Programming Problems

In this paper, we study a gradient-based neural network method for solving strictly convex quadratic programming (SCQP) problems. By converting the SCQP problem into a system of ordinary differential equation (ODE), we are able to show that the solution trajectory of this ODE tends to the set of stationary points of the original optimization problem. It is shown that the proposed neural network is stable in the sense of Lyapunov and can converge to an exact optimal solution of the original problem. It is also found that a larger scaling factor leads to a better convergence rate of the trajectory. The simulation results also show that the proposed neural network is feasible and efficient. The simulation results are very attractive.

[1]  Marina Fruehauf,et al.  Nonlinear Programming Analysis And Methods , 2016 .

[2]  N. Kalouptsidis.  Signal Processing Systems: Theory and Design , 1997 .

[3]  Jun Wang,et al.  A recurrent neural network with exponential convergence for solving convex quadratic program and related linear piecewise equations , 2004, Neural Networks.

[4]  Alfio Quarteroni,et al.  Numerical Mathematics (Texts in Applied Mathematics) , 2006 .

[5]  Leon O. Chua,et al.  Neural networks for nonlinear programming , 1988 .

[6]  Sunil K. Agrawal,et al.  Optimization of Dynamic Systems , 1999 .

[7]  Qingshan Liu,et al.  A One-Layer Recurrent Neural Network With a Discontinuous Hard-Limiting Activation Function for Quadratic Programming , 2008, IEEE Transactions on Neural Networks.

[8]  Jein-Shan Chen,et al.  A semismooth Newton method for SOCCPs based on a one-parametric class of SOC complementarity functions , 2010, Comput. Optim. Appl..

[9]  Jun Wang,et al.  A one-layer recurrent neural network for support vector machine learning , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[10]  Alireza Nazemi.  A dynamical model for solving degenerate quadratic minimax problems with constraints , 2011, J. Comput. Appl. Math..

[11]  Stefen Hui,et al.  On solving constrained optimization problems with neural networks: a penalty method approach , 1993, IEEE Trans. Neural Networks.

[12]  Nicole Propst.  Signal Processing Systems Theory And Design , 2016 .

[13]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[14]  Alireza Nazemi,et al.  A capable neural network model for solving the maximum flow problem , 2012, J. Comput. Appl. Math..

[15]  Paul D. Domich,et al.  An interior point method for general large-scale quadratic programming problems , 1996, Ann. Oper. Res..

[16]  Liwei Zhang,et al.  A globally convergent method based on Fischer-Burmeister operators for solving second-order cone constrained variational inequality problems , 2009, Comput. Math. Appl..

[17]  Yi Shen,et al.  A Modified Projection Neural Network for Linear Variational Inequalities and Quadratic Optimization Problems , 2009, ISNN.

[18]  Natashia Boland,et al.  A dual-active-set algorithm for positive semi-definite quadratic programming , 1997, Math. Program..

[19]  Defeng Sun,et al.  Strong Semismoothness of the Fischer-Burmeister SDC and SOC Complementarity Functions , 2005, Math. Program..

[20]  高等学校計算数学学報編輯委員会編.  高等学校計算数学学報 = Numerical mathematics , 1979 .

[21]  Jun Wang,et al.  A General Projection Neural Network for Solving Monotone Variational Inequalities and Related Optimization Problems , 2004 .

[22]  Jinde Cao,et al.  Global exponential stability of discrete-time recurrent neural network for solving quadratic programming problems subject to linear constraints , 2011, Neurocomputing.

[23]  Jinde Cao,et al.  A feedback neural network for solving convex constraint optimization problems , 2008, Appl. Math. Comput..

[24]  Francisco Facchinei,et al.  A smoothing method for mathematical programs with equilibrium constraints , 1999, Math. Program..

[25]  Jun Wang,et al.  A recurrent neural network for solving linear projection equations , 2000, Neural Networks.

[26]  Huang Yuancan,et al.  A Novel Method to Handle Inequality Constraints for Convex Programming Neural Network , 2002, Neural Processing Letters.

[27]  Katta G. Murty,et al.  Nonlinear Programming Theory and Algorithms , 2007, Technometrics.

[28]  Jein-Shan Chen,et al.  Neural networks for solving second-order cone constrained variational inequality problem , 2012, Comput. Optim. Appl..

[29]  Michael A. Shanblatt,et al.  Linear and quadratic programming neural network analysis , 1992, IEEE Trans. Neural Networks.

[30]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[31]  Jun Wang,et al.  A projection neural network and its application to constrained optimization problems , 2002 .

[32]  N. G. Parke,et al.  Ordinary Differential Equations. , 1958 .

[33]  Jun Wang,et al.  Primal neural networks for solving convex quadratic programs , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[34]  A. Nazemi,et al.  An efficient dynamic model for solving the shortest path problem , 2013 .

[35]  Jinde Cao,et al.  A simple and high performance neural network for quadratic programming problems , 2001, Appl. Math. Comput..

[36]  Yan Zhao,et al.  A continuous-time recurrent neural network for real-time support vector regression , 2013, 2013 IEEE Symposium on Computational Intelligence in Control and Automation (CICA).

[37]  Gerardo Toraldo,et al.  On the Solution of Large Quadratic Programming Problems with Bound Constraints , 1991, SIAM J. Optim..

[38]  Alaeddin Malek,et al.  Image fusion algorithms for color and gray level images based on LCLS method and novel artificial neural network , 2010, Neurocomputing.

[39]  Gang Feng,et al.  Solving convex quadratic programming problems by an modified neural network with exponential convergence , 2003, International Conference on Neural Networks and Signal Processing, 2003. Proceedings of the 2003.

[40]  Alireza Nazemi.  A dynamic system model for solving convex nonlinear optimization problems , 2012 .

[41]  Sheng-Long Hu,et al.  Properties of a family of generalized NCP-functions and a derivative free algorithm for complementarity problems , 2009 .

[42]  Youshen Xia,et al.  An improved neural network for convex quadratic optimization with application to real-time beamforming , 2005, Neurocomputing.

[43]  Insley B. Pyne,et al.  Linear programming on an electronic analogue computer , 1956, Transactions of the American Institute of Electrical Engineers, Part I: Communication and Electronics.

[44]  Jun Wang,et al.  A general projection neural network for solving monotone variational inequalities and related optimization problems , 2004, IEEE Transactions on Neural Networks.

[45]  You-Ping Chen,et al.  An Improved Neural Network for Solving Optimization of Quadratic Programming Problems , 2006, 2006 International Conference on Machine Learning and Cybernetics.

[46]  Davide Anguita,et al.  Improved neural network for SVM learning , 2002, IEEE Trans. Neural Networks.

[47]  Wei Bian,et al.  A project neural network for solving degenerate convex quadratic program , 2007, Neurocomputing.

[48]  Mokhtar S. Bazaraa,et al.  Nonlinear Programming: Theory and Algorithms, 3/E. , 2019 .

[49]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[50]  J. Hale,et al.  Ordinary Differential Equations , 2019, Fundamentals of Numerical Mathematics for Physicists and Engineers.

[51]  Liwei Zhang,et al.  An Augmented Lagrangian Method for a Class of Inverse Quadratic Programming Problems , 2009 .