Accelerated Block-coordinate Relaxation for Regularized Optimization

We discuss minimization of a smooth function to which is added a separable regularization function that induces structure in the solution. A block-coordinate relaxation approach with proximal linearized subproblems yields convergence to critical points, while identification of the optimal manifold (under a nondegeneracy condition) allows acceleration techniques to be applied on a reduced space. The work is motivated by experience with an algorithm for regularized logistic regression, and computational results for the algorithm on problems of this type are presented.

[1]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[2]  C. Hsiung.  A first course in differential geometry , 1981 .

[3]  Stephen J. Wright.  Identifiable Surfaces in Constrained Optimization , 1993 .

[4]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[5]  Sergey Bakin,et al.  Adaptive regression and model selection in data mining problems , 1999 .

[6]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[7]  Adrian S. Lewis,et al.  Active Sets, Nonsmoothness, and Sensitivity , 2002, SIAM J. Optim..

[8]  S. Sathiya Keerthi,et al.  A simple and efficient algorithm for gene selection using sparse logistic regression , 2003, Bioinform..

[9]  A. Lewis,et al.  Identifying active constraints via partial smoothness and prox-regularity , 2003 .

[10]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[11]  Stephen J. Wright,et al.  Simultaneous Variable Selection , 2005, Technometrics.

[12]  Stephen J. Wright,et al.  LASSO-Patternsearch Algorithm with Application to Ophthalmology Data , 2006 .

[13]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[14]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[15]  Stephen P. Boyd,et al.  An Interior-Point Method for Large-Scale l1-Regularized Logistic Regression , 2007, J. Mach. Learn. Res..

[16]  Yin Zhang,et al.  Fixed-Point Continuation for l1-Minimization: Methodology and Convergence , 2008, SIAM J. Optim..

[17]  Grace Wahba,et al.  LASSO-Patternsearch algorithm with application to ophthalmology and genomic data. , 2006, Statistics and its interface.

[18]  P. Tseng,et al.  Block-Coordinate Gradient Descent Method for Linearly Constrained Nonsmooth Separable Optimization , 2009 .

[19]  Yoram Singer,et al.  Efficient projections onto the l1-ball for learning in high dimensions , 2008, ICML '08.

[20]  P. Bühlmann,et al.  The group lasso for logistic regression , 2008 .

[21]  Stephen J. Wright,et al.  Sparse Reconstruction by Separable Approximation , 2008, IEEE Transactions on Signal Processing.

[22]  Paul Tseng.  Further Results on Stable Recovery of Sparse Overcomplete Representations in the Presence of Noise , 2009, IEEE Transactions on Information Theory.

[23]  Warren Hare,et al.  A proximal method for identifying active manifolds , 2009, Comput. Optim. Appl..

[24]  Paul Tseng,et al.  A coordinate gradient descent method for nonsmooth separable minimization , 2008, Math. Program..

[25]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[26]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[27]  Paul Tseng,et al.  A coordinate gradient descent method for linearly constrained smooth optimization and support vector machines training , 2010, Comput. Optim. Appl..

[28]  Gillian M. Chin,et al.  On the Use of Stochastic Hessian Information in Unconstrained Optimization , 2010 .

[29]  Wotao Yin,et al.  A Fast Hybrid Algorithm for Large-Scale l1-Regularized Logistic Regression , 2010, J. Mach. Learn. Res..

[30]  Paul Tseng,et al.  Approximation accuracy, gradient methods, and error bound for structured convex optimization , 2010, Math. Program..

[31]  Laurent El Ghaoui,et al.  Safe Feature Elimination for the LASSO and Sparse Supervised Learning Problems , 2010, 1009.4219.

[32]  Guo-Xun Yuan.  A Comparison of Optimization Methods for Large-scale L 1-regularized Linear Classification , 2010 .

[33]  W. L. Hare,et al.  Identifying Active Manifolds in Regularization Problems , 2011, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.