MPGL: An Efficient Matching Pursuit Method for Generalized LASSO

Unlike traditional LASSO enforcing sparsity on the variables, Generalized LASSO (GL) enforces sparsity on a linear transformation of the variables, gaining flexibility and success in many applications. However, many existing GL algorithms do not scale up to high-dimensional problems, and/or only work well for a specific choice of the transformation. We propose an efficient Matching Pursuit Generalized LASSO (MPGL) method, which overcomes these issues, and is guaranteed to converge to a global optimum. We formulate the GL problem as a convex quadratic constrained linear programming (QCLP) problem and tailor-make a cutting plane method. More specifically, our MPGL iteratively activates a subset of nonzero elements of the transformed variables, and solves a subproblem involving only the activated elements thus gaining significant speed-up. Moreover, MPGL is less sensitive to the choice of the trade-off hyper-parameter between data fitting and regularization, and mitigates the longstanding hyper-parameter tuning issue in many existing methods. Experiments demonstrate the superior efficiency and accuracy of the proposed method over the state-of-the-arts in both classification and image processing tasks. Introduction Learning with sparsity-inducing norms has gained much success in many applications including medical data analysis (Tibshirani and Wang 2008), image processing (Rudin, Osher, and Fatemi 1992), feature selection (Tan, Tsang, and Wang 2014) and so on. One efficient way to enforce sparsity on the variables is to use the `1-norm as LASSO (Tibshirani 1996) instead of the `0-norm. Since then, many methods have been proposed to enforce some additional constraints (Huang, Zhang, and Metaxas 2011; Kim and Xing 2010; Tibshirani et al. 2011) to improve the results. A group of methods among them is called generalized LASSO (Tibshirani et al. 2011), which promotes the sparsity of the variables after a linear transformation (Liu, Yuan, and Ye 2013) instead of the variables themselves. The choice of such a transformation represents the property of the variables to be desired, and often depends on the application. Generalized LASSO. Let x ∈ R denote the target variable and D ∈ Rl×n be a linear transformation operator. A natural The first two authors contributed equally. Copyright c © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. way to seek x with sparsity on Dx is as follows (Liu, Yuan, and Ye 2013), min x f(x) + λ‖Dx‖0, (1) where f : R → R is a loss function (sometimes known as data fitting term) depending on the application, ‖·‖0 denotes the `0-norm regularizer, and λ ≥ 0 is known as the trade-off hyper-parameter between the data fitting and the regularization. By letting A ∈ Rm×n be a designing matrix, y ∈ R be a response vector, n ∈ R be a vector of Gaussian noise, and assuming a linear regression model y = Ax + n, a typical choice of f is f(x) = 1 2‖y −Ax‖ 2 2, which will be used throughout the rest of the paper. Since problem (1) is NP-hard, a convex relaxation is widely used: