An inexact successive quadratic approximation method for L-1 regularized optimization

We study a Newton-like method for the minimization of an objective function that is the sum of a smooth convex function and an l-1 regularization term. This method, which is sometimes referred to in the literature as a proximal Newton method, computes a step by minimizing a piecewise quadratic model of the objective function. In order to make this approach efficient in practice, it is imperative to perform this inner minimization inexactly. In this paper, we give inexactness conditions that guarantee global convergence and that can be used to control the local rate of convergence of the iteration. Our inexactness conditions are based on a semi-smooth function that represents a (continuous) measure of the optimality conditions of the problem, and that embodies the soft-thresholding iteration. We give careful consideration to the algorithm employed for the inner minimization, and report numerical results on two test sets originating in machine learning.

[1]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[2]  R. Dembo,et al.  INEXACT NEWTON METHODS , 1982 .

[3]  Jorge Nocedal,et al.  Representations of quasi-Newton matrices and their use in limited memory methods , 1994, Math. Program..

[4]  Jean-Yves Audibert.  Optimization for Machine Learning , 1995 .

[5]  Michael Patriksson,et al.  Cost Approximation: A Unified Framework of Descent Algorithms for Nonlinear Programs , 1998, SIAM J. Optim..

[6]  M. Patriksson.  Nonlinear Programming and Variational Inequality Problems: A Unified Approach , 1998 .

[7]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[8]  M. Patriksson.  Nonlinear Programming and Variational Inequality Problems , 1999 .

[9]  F. Facchinei,et al.  Finite-Dimensional Variational Inequalities and Complementarity Problems , 2003 .

[10]  Leonhard Held,et al.  Gaussian Markov Random Fields: Theory and Applications , 2005 .

[11]  Alexandre d'Aspremont,et al.  Convex optimization techniques for fitting sparse Gaussian graphical models , 2006, ICML.

[12]  Jianfeng Gao,et al.  Scalable training of L1-regularized log-linear models , 2007, ICML '07.

[13]  Alexandre d'Aspremont,et al.  Model Selection Through Sparse Max Likelihood Estimation Model Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data , 2022 .

[14]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[15]  Stephen J. Wright,et al.  Sparse Reconstruction by Separable Approximation , 2008, IEEE Transactions on Signal Processing.

[16]  Lu Li,et al.  An inexact interior point method for L1-regularized sparse covariance selection , 2010, Math. Program. Comput..

[17]  Emmanuel J. Candès,et al.  Templates for convex cone problems with applications to sparse signal recovery , 2010, Math. Program. Comput..

[18]  Pradeep Ravikumar,et al.  Sparse inverse covariance matrix estimation using quadratic approximation , 2011, MLSLP.

[19]  Saverio Salzo,et al.  Inexact and accelerated proximal point algorithms , 2011 .

[20]  Mark W. Schmidt,et al.  Convergence Rates of Inexact Proximal-Gradient Methods for Convex Optimization , 2011, NIPS.

[21]  Jorge Nocedal,et al.  Newton-Like Methods for Sparse Inverse Covariance Estimation , 2012, NIPS.

[22]  Jorge Nocedal,et al.  Sample size selection in optimization methods for machine learning , 2012, Math. Program..

[23]  R. Tyrrell Rockafellar,et al.  Convergence of inexact Newton methods for generalized equations , 2013, Math. Program..

[24]  Katya Scheinberg,et al.  Complexity of Inexact Proximal Newton methods , 2013, ArXiv.

[25]  Michael A. Saunders,et al.  Proximal Newton-Type Methods for Minimizing Composite Functions , 2012, SIAM J. Optim..

[26]  Michael Ulbrich,et al.  A Semismooth Newton Method with Multidimensional Filter Globalization for l1-Optimization , 2014, SIAM J. Optim..