Neural Network for Nonsmooth, Nonconvex Constrained Minimization Via Smooth Approximation

A neural network based on smoothing approximation is presented for a class of nonsmooth, nonconvex constrained optimization problems, where the objective function is nonsmooth and nonconvex, the equality constraint functions are linear and the inequality constraint functions are nonsmooth, convex. This approach can find a Clarke stationary point of the optimization problem by following a continuous path defined by a solution of an ordinary differential equation. The global convergence is guaranteed if either the feasible set is bounded or the objective function is level bounded. Specially, the proposed network does not require: 1) the initial point to be feasible; 2) a prior penalty parameter to be chosen exactly; 3) a differential inclusion to be solved. Numerical experiments and comparisons with some existing algorithms are presented to illustrate the theoretical results and show the efficiency of the proposed network.

[1]  Jun Wang,et al.  A Novel Recurrent Neural Network for Solving Nonlinear Optimization Problems With Inequality Constraints , 2008, IEEE Transactions on Neural Networks.

[2]  R. Tibshirani,et al.  Regression shrinkage and selection via the lasso: a retrospective , 2011 .

[3]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .

[4]  Xiaojun Chen.  SMOOTHING METHODS FOR COMPLEMENTARITY PROBLEMS AND THEIR APPLICATIONS : A SURVEY , 2000 .

[5]  Michael L. Overton,et al.  A Sequential Quadratic Programming Algorithm for Nonconvex, Nonsmooth Constrained Optimization , 2012, SIAM J. Optim..

[6]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.

[7]  P. Bahr,et al.  Sampling: Theory and Applications , 2020, Applied and Numerical Harmonic Analysis.

[8]  Mauro Forti,et al.  Convergence of Neural Networks for Programming Problems via a Nonsmooth Łojasiewicz Inequality , 2006, IEEE Transactions on Neural Networks.

[9]  Takayuki Yamada,et al.  Dynamic system identification using neural networks , 1993, IEEE Trans. Syst. Man Cybern..

[10]  Mauro Forti.  M-matrices and global convergence of discontinuous neural networks , 2007, Int. J. Circuit Theory Appl..

[11]  Reuven Y. Rubinstein,et al.  Nondifferentiable optimization via smooth approximation: General analytical approach , 1992, Ann. Oper. Res..

[12]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[13]  David Betounes.  Differential Equations: Theory and Applications , 2001 .

[14]  Mert Gürbüzbalaban,et al.  On Nesterov's Nonsmooth Chebyshev-Rosenbrock Functions , 2012 .

[15]  Edwin K. P. Chong,et al.  An analysis of a class of neural networks for solving linear programming problems , 1999, IEEE Trans. Autom. Control..

[16]  Xiaojun Chen,et al.  Smoothing Projected Gradient Method and Its Application to Stochastic Linear Complementarity Problems , 2009, SIAM J. Optim..

[17]  R. Chartrand,et al.  Restricted isometry properties and nonconvex compressive sensing , 2007 .

[18]  Xiaojun Chen,et al.  Smoothing methods for nonsmooth, nonconvex minimization , 2012, Math. Program..

[19]  F. Clarke.  Optimization And Nonsmooth Analysis , 1983 .

[20]  A LeonO.EtAl.Chu,et al.  Linear and nonlinear circuits , 2014 .

[21]  Xiaoqi Yang,et al.  Deriving sufficient conditions for global asymptotic stability of delayed neural networks via nonsmooth analysis , 2004, IEEE Transactions on Neural Networks.

[22]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[23]  Xiaojun Chen,et al.  Smoothing Neural Network for Constrained Non-Lipschitz Optimization With Applications , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[24]  Mauro Forti,et al.  Generalized neural network for nonsmooth nonlinear programming problems , 2004, IEEE Transactions on Circuits and Systems I: Regular Papers.

[25]  Qingshan Liu,et al.  A one-layer recurrent neural network for constrained pseudoconvex optimization and its application for dynamic portfolio optimization , 2012, Neural Networks.

[26]  Zidong Wang,et al.  State Estimation for Coupled Uncertain Stochastic Networks With Missing Measurements and Time-Varying Delays: The Discrete-Time Case , 2009, IEEE Transactions on Neural Networks.

[27]  Peter Bühlmann.  Regression shrinkage and selection via the Lasso: a retrospective (Robert Tibshirani): Comments on the presentation , 2011 .

[28]  Damiana Lazzaro,et al.  Nonlinear Filtering for Sparse Signal Recovery From Incomplete Measurements , 2009, IEEE Transactions on Signal Processing.

[29]  Johan A. K. Suykens,et al.  Application of a Smoothing Technique to Decomposition in Convex Optimization , 2008, IEEE Transactions on Automatic Control.

[30]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[31]  Jun Wang,et al.  Convergence Analysis of a Class of Nonsmooth Gradient Systems , 2008, IEEE Transactions on Circuits and Systems I: Regular Papers.

[32]  Xiaojun Chen,et al.  Minimizing the Condition Number of a Gram Matrix , 2011, SIAM J. Optim..

[33]  Xiaojun Chen,et al.  Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization , 2015, Math. Program..

[34]  Qingshan Liu,et al.  Finite-Time Convergent Recurrent Neural Network With a Hard-Limiting Activation Function for Constrained Optimization With Piecewise-Linear Objective Functions , 2011, IEEE Transactions on Neural Networks.

[35]  Mauro Forti,et al.  Lyapunov Method and Convergence of the Full-Range Model of CNNs , 2008, IEEE Transactions on Circuits and Systems I: Regular Papers.

[36]  Jun Wang,et al.  A One-Layer Recurrent Neural Network for Constrained Nonsmooth Optimization , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[37]  Xiaojun Chen,et al.  Worst-Case Complexity of Smoothing Quadratic Regularization Methods for Non-Lipschitzian Optimization , 2013, SIAM J. Optim..

[38]  Long Cheng,et al.  Recurrent Neural Network for Non-Smooth Convex Optimization Problems With Application to the Identification of Genetic Regulatory Networks , 2011, IEEE Transactions on Neural Networks.

[39]  P. H. Quang,et al.  Generalized Convexity of Functions and Generalized Monotonicity of Set-Valued Maps , 1997 .

[40]  Wei Bian,et al.  Subgradient-Based Neural Networks for Nonsmooth Nonconvex Optimization Problems , 2009, IEEE Transactions on Neural Networks.