A modified local quadratic approximation algorithm for penalized optimization problems

In this paper, we propose an optimization algorithm called the modified local quadratic approximation algorithm for minimizing various ? 1 -penalized convex loss functions. The proposed algorithm iteratively solves ? 1 -penalized local quadratic approximations of the loss function, and then modifies the solution whenever it fails to decrease the original ? 1 -penalized loss function. As an extension, we construct an algorithm for minimizing various nonconvex penalized convex loss functions by combining the proposed algorithm and convex concave procedure, which can be applied to most nonconvex penalty functions such as the smoothly clipped absolute deviation and minimax concave penalty functions. Numerical studies show that the algorithm is stable and fast for solving high dimensional penalized optimization problems.

[1]  Alan L. Yuille,et al.  The Concave-Convex Procedure , 2003, Neural Computation.

[2]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[3]  Chia-Hua Ho,et al.  An improved GLMNET for l1-regularized logistic regression , 2011, J. Mach. Learn. Res..

[4]  U. Alon,et al.  Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[5]  S. Dudoit,et al.  Comparison of Discrimination Methods for the Classification of Tumors Using Gene Expression Data , 2002 .

[6]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[7]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[8]  B. Lindsay,et al.  Monotonicity of quadratic-approximation algorithms , 1988 .

[9]  Yi Lin,et al.  Statistical Properties and Adaptive Tuning of Support Vector Machines , 2002, Machine Learning.

[10]  Jian Huang,et al.  COORDINATE DESCENT ALGORITHMS FOR NONCONVEX PENALIZED REGRESSION, WITH APPLICATIONS TO BIOLOGICAL FEATURE SELECTION. , 2011, The annals of applied statistics.

[11]  Steve R. Gunn,et al.  Structural Modelling with Sparse Kernels , 2002, Machine Learning.

[12]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[13]  Meta M. Voelker,et al.  Variable Selection and Model Building via Likelihood Basis Pursuit , 2004 .

[14]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[15]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[16]  R. Tibshirani,et al.  PATHWISE COORDINATE OPTIMIZATION , 2007, 0708.1485.

[17]  M. R. Osborne,et al.  A new approach to variable selection in least squares problems , 2000 .

[18]  James A. Piazza.  The Opium Trade and Patterns of Terrorism in the Provinces of Afghanistan: An Empirical Analysis , 2012 .

[19]  Yongdai Kim,et al.  Smoothly Clipped Absolute Deviation on High Dimensions , 2008 .

[20]  Kim,et al.  A gradient-based optimization algorithm for LASSO , 2008 .

[21]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[22]  P. Wriggers.  Nonlinear finite element analysis of solids and structures , 1998 .

[23]  S. Rosset,et al.  Piecewise linear regularized solution paths , 2007, 0708.2197.

[24]  Mee Young Park,et al.  L1‐regularization path algorithm for generalized linear models , 2007 .

[25]  Lawrence Carin,et al.  Sparse multinomial logistic regression: fast algorithms and generalization bounds , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Sunghoon Kwon,et al.  Multiclass sparse logistic regression for classification of multiple cancer types using gene expression data , 2006, Comput. Stat. Data Anal..