A proximal method for composite minimization

We consider minimization of functions that are compositions of convex or prox-regular functions (possibly extended-valued) with smooth vector functions. A wide variety of important optimization problems fall into this framework. We describe an algorithmic framework based on a subproblem constructed from a linearized approximation to the objective and a regularization term. Properties of local solutions of this subproblem underlie both a global convergence result and an identification property of the active manifold containing the solution of the original problem. Preliminary computational results on both convex and nonconvex examples are promising.

[1]  B. Martinet,et al.  R'egularisation d''in'equations variationnelles par approximations successives , 1970 .

[2]  B. Martinet.  Brève communication. Régularisation d'inéquations variationnelles par approximations successives , 1970 .

[3]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[4]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[5]  M. Fukushima,et al.  A generalized proximal point algorithm for certain non-convex minimization problems , 1981 .

[6]  J. Spingarn.  Submonotone mappings and the proximal point algorithm , 1982 .

[7]  James V. Burke,et al.  Descent methods for composite nondifferentiable optimization problems , 1985, Math. Program..

[8]  Ya-Xiang Yuan,et al.  Conditions for convergence of trust region algorithms for nonsmooth optimization , 1985, Math. Program..

[9]  Ya-Xiang Yuan,et al.  On the superlinear convergence of a trust region algorithm for nonsmooth optimization , 1985, Math. Program..

[10]  J. J. Moré,et al.  On the identification of active constraints , 1988 .

[11]  Roger Fletcher,et al.  Nonlinear programming and nonsmooth optimization by successive linear programming , 1989, Math. Program..

[12]  C. Reinsch,et al.  An Analysis of Two Algorithms for Shape-Preserving Cubic Spline Interpolation , 1989 .

[13]  J. Burke.  On identification of active constraints II: the nonconvex case , 1990 .

[14]  Stephen J. Wright.  Convergence of an inexact algorithm for composite nonsmooth optimization , 1990 .

[15]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[16]  Stephen J. Wright.  Identifiable Surfaces in Constrained Optimization , 1993 .

[17]  R. Rockafellar,et al.  Prox-regular functions in variational analysis , 1996 .

[18]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[19]  Alexander Kaplan,et al.  Proximal Point Methods and Nonconvex Optimization , 1998, J. Glob. Optim..

[20]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[21]  C. Lemaréchal,et al.  THE U -LAGRANGIAN OF A CONVEX FUNCTION , 1996 .

[22]  O. Mangasarian.  Minimum-support solutions of polyhedral concave programs * , 1999 .

[23]  R. Rockafellar,et al.  Local differentiability of distance functions , 2000 .

[24]  J. Frédéric Bonnans,et al.  Perturbation Analysis of Optimization Problems , 2000, Springer Series in Operations Research.

[25]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[26]  Adam B. Levy.  Lipschitzian Multifunctions and a Lipschitzian Inverse Mapping Theorem , 2001, Math. Oper. Res..

[27]  R. Rockafellar,et al.  The radius of metric regularity , 2002 .

[28]  Adrian S. Lewis,et al.  Active Sets, Nonsmoothness, and Sensitivity , 2002, SIAM J. Optim..

[29]  R. Mi,et al.  Proximal Points are on the Fast Track , 2002 .

[30]  Teemu Pennanen,et al.  Local Convergence of the Proximal Point Algorithm and Multiplier Methods Without Monotonicity , 2002, Math. Oper. Res..

[31]  Alfredo N. Iusem,et al.  Inexact Variants of the Proximal Point Algorithm without Monotonicity , 2002, SIAM J. Optim..

[32]  Alexander Shapiro,et al.  On a Class of Nonsmooth Composite Functions , 2003, Math. Oper. Res..

[33]  A. Lewis,et al.  Identifying active constraints via partial smoothness and prox-regularity , 2003 .

[34]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[35]  Patrick L. Combettes,et al.  Proximal Methods for Cohypomonotone Operators , 2004, SIAM J. Control. Optim..

[36]  Jérôme Malick,et al.  Newton methods for nonsmooth convex minimization: connections among -Lagrangian, Riemannian Newton and SQP methods , 2005, Math. Program..

[37]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[38]  Robert Mifflin,et al.  A -algorithm for convex minimization , 2005, Math. Program..

[39]  Nicholas I. M. Gould,et al.  On the Convergence of Successive Linear-Quadratic Programming Algorithms , 2005, SIAM J. Optim..

[40]  A. Daniilidis,et al.  Geometrical interpretation of the predictor-corrector type algorithms in structured optimization problems , 2006 .

[41]  B. Mordukhovich.  Variational analysis and generalized differentiation , 2006 .

[42]  E.J. Candes.  Compressive Sampling , 2022 .

[43]  Stephen J. Wright,et al.  LASSO-Patternsearch Algorithm with Application to Ophthalmology Data , 2006 .

[44]  Xiaodong Lin,et al.  Gene expression Gene selection using support vector machines with non-convex penalty , 2005 .

[45]  M. Friedlander,et al.  A Filter Active-Set Trust-Region Method , 2007 .

[46]  Yin Zhang,et al.  Fixed-Point Continuation for l1-Minimization: Methodology and Convergence , 2008, SIAM J. Optim..

[47]  A. Kruger,et al.  Metric regularity and systems of generalized equations , 2008 .

[48]  Grace Wahba,et al.  LASSO-Patternsearch algorithm with application to ophthalmology and genomic data. , 2006, Statistics and its interface.

[49]  Marc E. Pfetsch,et al.  Exact and Approximate Sparse Solutions of Underdetermined Linear Equations , 2008, SIAM J. Sci. Comput..

[50]  Stephen J. Wright,et al.  Sparse reconstruction by separable approximation , 2009, IEEE Trans. Signal Process..

[51]  Boris Polyak,et al.  B.S. Mordukhovich. Variational Analysis and Generalized Differentiation. I. Basic Theory, II. Applications , 2009 .

[52]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[53]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[54]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[55]  Yin Zhang,et al.  A Fast Algorithm for Sparse Reconstruction Based on Shrinkage, Subspace Optimization, and Continuation , 2010, SIAM J. Sci. Comput..

[56]  Adrian S. Lewis,et al.  Generic Optimality Conditions for Semialgebraic Convex Programs , 2011, Math. Oper. Res..

[57]  R D Zimmerman,et al.  MATPOWER: Steady-State Operations, Planning, and Analysis Tools for Power Systems Research and Education , 2011, IEEE Transactions on Power Systems.

[58]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2008, Found. Comput. Math..

[59]  Wotao Yin,et al.  On the convergence of an active-set method for ℓ1 minimization , 2012, Optim. Methods Softw..

[60]  Claudia A. Sagastizábal,et al.  Composite proximal bundle method , 2013, Math. Program..

[61]  Stephen J. Wright,et al.  An S$\ell_1$LP-Active Set Approach for Feasibility Restoration in Power Systems , 2014, 1405.0322.

[62]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.

[63]  Guanghui Lan,et al.  Bundle-level type methods uniformly optimal for smooth and nonsmooth convex optimization , 2013, Mathematical Programming.