L1 LASSO Modeling and Its Bayesian Inference

A new iterative procedure for solving regression problems with the so-called LASSO penalty [1] is proposed by using generative Bayesian modeling and inference. The algorithm produces the anticipated parsimonious or sparse regression models that generalize well on unseen data. The proposed algorithm is quite robust and there is no need to specify any model hyperparameters. A comparison with state-of-the-art methods for constructing sparse regression models such as the relevance vector machine (RVM) and the local regularization assisted orthogonal least squares regression (LROLS) is given.

[1]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[2]  Johan A. K. Suykens,et al.  Bayesian input selection for nonlinear regression with LS-SVMS , 2003 .

[3]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[4]  Mehmet A. Orgun,et al.  AI 2007: Advances in Artificial Intelligence, 20th Australian Joint Conference on Artificial Intelligence, Gold Coast, Australia, December 2-6, 2007, Proceedings , 2007, Australian Conference on Artificial Intelligence.

[5]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[6]  Junbin Gao,et al.  Robust L1 Principal Component Analysis and Its Bayesian Variational Inference , 2008, Neural Computation.

[7]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[8]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[9]  G. Horváth,et al.  A WEIGHTED GENERALIZED LS-SVM , 2003 .

[10]  Nicolai Meinshausen,et al.  Relaxed Lasso , 2007, Comput. Stat. Data Anal..

[11]  G. Horváth,et al.  A generalised LS-SVM , 2003 .

[12]  B. J. de Kruif,et al.  Support-vector-based least squares for learning non-linear dynamics , 2002, CDC.

[13]  S. A. Billings,et al.  The identification of linear and non-linear models of a turbocharged automotive diesel engine , 1989 .

[14]  Massimiliano Pontil,et al.  On the Noise Model of Support Vector Machines Regression , 2000, ALT.

[15]  T. Hesterberg,et al.  Least angle and ℓ1 penalized regression: A review , 2008, 0802.0964.

[16]  Bob McKay,et al.  AI 2002: Advances in Artificial Intelligence , 2002, Lecture Notes in Computer Science.

[17]  George Eastman House,et al.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001 .

[18]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[19]  Sheng Chen,et al.  Local regularization assisted orthogonal least squares regression , 2006, Neurocomputing.

[20]  Junbin Gao,et al.  Adapting Kernels by Variational Approach in SVM , 2002, Australian Joint Conference on Artificial Intelligence.

[21]  Junbin Gao,et al.  Mixture of the Robust L1 Distributions and Its Applications , 2007, Australian Conference on Artificial Intelligence.

[22]  R. Harrison,et al.  Support Vector Machines for System Identification , 1998 .