Optimality and Complexity for Constrained Optimization Problems with Nonconvex Regularization

In this paper, we consider a class of constrained optimization problems where the feasible set is a general closed convex set, and the objective function has a nonsmooth, nonconvex regularizer. Such a regularizer includes widely used SCAD, MCP, logistic, fraction, hard thresholding, and non-Lipschitz Lp penalties as special cases. Using the theory of the generalized directional derivative and the tangent cone, we derive a first order necessary optimality condition for local minimizers of the problem, and define the generalized stationary point of it. We show that the generalized stationary point is the Clarke stationary point when the objective function is Lipschitz continuous at this point, and satisfies the existing necessary optimality conditions when the objective function is not Lipschitz continuous at this point. Moreover, we prove the consistency between the generalized directional derivative and the limit of the classic directional derivatives associated with the smoothing function. Finally, we es...

[1]  Michael L. Overton,et al.  A Sequential Quadratic Programming Algorithm for Nonconvex, Nonsmooth Constrained Optimization , 2012, SIAM J. Optim..

[2]  J. Burke,et al.  Gradient Consistency for Integral-convolution Smoothing Functions , 2013 .

[3]  Raymond H. Chan,et al.  Half-Quadratic Algorithm for ℓp - ℓq Problems with Applications to TV-ℓ1 Image Restoration and Compressive Sensing , 2011, Efficient Algorithms for Global Optimization Methods in Computer Vision.

[4]  Xiaojun Chen,et al.  Linearly Constrained Non-Lipschitz Optimization for Image Restoration , 2015, SIAM J. Imaging Sci..

[5]  I. Daubechies,et al.  Sparse and stable Markowitz portfolios , 2007, Proceedings of the National Academy of Sciences.

[6]  J. Jahn.  Introduction to the Theory of Nonlinear Optimization , 1994 .

[7]  James V. Burke,et al.  Epi-convergent Smoothing with Applications to Convex Composite Functions , 2012, SIAM J. Optim..

[8]  R. T. Rockafellar,et al.  The Generic Nature of Optimality Conditions in Nonlinear Programming , 1979, Math. Oper. Res..

[9]  LuZhaosong.  Iterative reweighted minimization methods for $$l_p$$lp regularized unconstrained nonlinear programming , 2014 .

[10]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[11]  J. Aubin,et al.  Differential inclusions set-valued maps and viability theory , 1984 .

[12]  R. Chartrand,et al.  Restricted isometry properties and nonconvex compressive sensing , 2007 .

[13]  Alfred Auslender,et al.  How to deal with the unbounded in optimization: Theory and algorithms , 1997, Math. Program..

[14]  Adrian S. Lewis,et al.  Approximating Subdifferentials by Random Sampling of Gradients , 2002, Math. Oper. Res..

[15]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[16]  Xiaojun Chen,et al.  Optimality Conditions and a Smoothing Trust Region Newton Method for NonLipschitz Optimization , 2013, SIAM J. Optim..

[17]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[18]  Cun-Hui Zhang,et al.  A group bridge approach for variable selection , 2009, Biometrika.

[19]  Balas K. Natarajan,et al.  Sparse Approximate Solutions to Linear Systems , 1995, SIAM J. Comput..

[20]  Y. Ye,et al.  Lower Bound Theory of Nonzero Entries in Solutions of ℓ2-ℓp Minimization , 2010, SIAM J. Sci. Comput..

[21]  P. Atzberger.  Introduction to Nonlinear Optimization , 2020, Linear Algebra and Optimization with Applications to Machine Learning.

[22]  Marc Teboulle,et al.  Smoothing and First Order Methods: A Unified Framework , 2012, SIAM J. Optim..

[23]  Jianqing Fan,et al.  Nonconcave penalized likelihood with a diverging number of parameters , 2004, math/0406466.

[24]  Boris Polyak,et al.  Constrained minimization methods , 1966 .

[25]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.

[26]  Shiqian Ma,et al.  A smoothing SQP framework for a class of composite Lq\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_q$$\end{documen , 2014, Mathematical Programming.

[27]  Yvonne Freeh,et al.  Interior Point Algorithms Theory And Analysis , 2016 .

[28]  Zhaosong Lu,et al.  Iterative reweighted minimization methods for $$l_p$$lp regularized unconstrained nonlinear programming , 2012, Math. Program..

[29]  Xiaojun Chen,et al.  Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization , 2015, Math. Program..

[30]  Hao Yin,et al.  Strong NP-Hardness Result for Regularized $L_q$-Minimization Problems with Concave Penalty Functions , 2015, ArXiv.

[31]  J. Horowitz,et al.  Asymptotic properties of bridge estimators in sparse high-dimensional regression models , 2008, 0804.0693.

[32]  NG MICHAELK.,et al.  NONCONVEX `P -REGULARIZATION AND BOX CONSTRAINED MODEL FOR IMAGE RESTORATION , 2011 .

[33]  E. Polak,et al.  Constrained Minimization Problems in Finite-Dimensional Spaces , 1966 .

[34]  Zhaoran Wang,et al.  OPTIMAL COMPUTATIONAL AND STATISTICAL RATES OF CONVERGENCE FOR SPARSE NONCONVEX LEARNING PROBLEMS. , 2013, Annals of statistics.

[35]  Mila Nikolova,et al.  Efficient Reconstruction of Piecewise Constant Images Using Nonsmooth Nonconvex Minimization , 2008, SIAM J. Imaging Sci..

[36]  Yinyu Ye,et al.  A note on the complexity of Lp minimization , 2011, Math. Program..

[37]  Wei Bian,et al.  Optimality Conditions and Complexity for Non-Lipschitz Constrained Optimization Problems , 2014 .

[38]  Mila Nikolova,et al.  Analysis of the Recovery of Edges in Images and Signals by Minimizing Nonconvex Regularized Least-Squares , 2005, Multiscale Model. Simul..

[39]  Jianqing Fan,et al.  Comments on «Wavelets in statistics: A review» by A. Antoniadis , 1997 .

[40]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[41]  Adrian S. Lewis,et al.  Convex Analysis And Nonlinear Optimization , 2000 .

[42]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[43]  Xiaojun Chen,et al.  Worst-Case Complexity of Smoothing Quadratic Regularization Methods for Non-Lipschitzian Optimization , 2013, SIAM J. Optim..

[44]  D. Bertsekas.  On the Goldstein-Levitin-Polyak gradient projection method , 1974, CDC 1974.

[45]  Xiaojun Chen,et al.  Complexity of unconstrained $$L_2-L_p$$ minimization , 2011, Math. Program..

[46]  Vijay V. Vazirani,et al.  Approximation Algorithms , 2001, Springer Berlin Heidelberg.

[47]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[48]  Charles Audet,et al.  Mesh Adaptive Direct Search Algorithms for Constrained Optimization , 2006, SIAM J. Optim..

[49]  Y. Ye,et al.  Sparse Portfolio Selection via Quasi-Norm Regularization , 2013, 1312.6350.

[50]  Xiaojun Chen,et al.  Non-Lipschitz $\ell_{p}$-Regularization and Box Constrained Model for Image Restoration , 2012, IEEE Transactions on Image Processing.

[51]  Zizhuo Wang,et al.  Complexity of Unconstrained L2-Lp Minimization , 2011 .

[52]  Wenjiang J. Fu,et al.  Asymptotics for lasso-type estimators , 2000 .

[53]  J. Hiriart-Urruty,et al.  Generalized Hessian matrix and second-order optimality conditions for problems withC1,1 data , 1984 .

[54]  Po-Ling Loh,et al.  Regularized M-estimators with nonconvexity: statistical and algorithmic theory for local optima , 2013, J. Mach. Learn. Res..

[55]  Xiaojun Chen,et al.  Smoothing methods for nonsmooth, nonconvex minimization , 2012, Math. Program..

[56]  F. Clarke.  Optimization And Nonsmooth Analysis , 1983 .

[57]  Ya-Xiang Yuan,et al.  Optimization Theory and Methods: Nonlinear Programming , 2010 .