Regression by L1 regularization of smart contrasts and sums (ROSCAS) beats PLS and elastic net in latent variable model

This paper proposes a regression method, ROSCAS, which regularizes smart contrasts and sums of regression coefficients by an L1 penalty. The contrasts and sums are based on the sample correlation matrix of the predictors and are suggested by a latent variable regression model. The contrasts express the idea that a priori correlated predictors should have similar coefficients. The method has excellent predictive performance in situations, where there are groups of predictors with each group representing an independent feature that influences the response. In particular, when the groups differ in size, ROSCAS can outperform LASSO, elastic net, partial least squares (PLS) and ridge regression by a factor of two or three in terms of mean squared error. In other simulation setups and on real data, ROSCAS performs competitively

[1]  L. Breiman.  Heuristics of instability and stabilization in model selection , 1996 .

[2]  W. Buist,et al.  Prediction of the percentage lean of pig carcasses with a small or a large number of instrumental carcass measurements - an illustration with HGP and Vision , 2006 .

[3]  Ron Wehrens,et al.  The pls Package: Principal Component and Partial Least Squares Regression in R , 2007 .

[4]  Lutgarde M. C. Buydens,et al.  Strategy for constructing robust multivariate calibration models , 1999 .

[5]  R. Tibshirani,et al.  Sparsity and smoothness via the fused lasso , 2005 .

[6]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[7]  M. R. Osborne,et al.  A new approach to variable selection in least squares problems , 2000 .

[8]  Svante Wold.  Discussion: PLS in Chemical Practice , 1993 .

[9]  S. Wold,et al.  The Collinearity Problem in Linear Regression. The Partial Least Squares (PLS) Approach to Generalized Inverses , 1984 .

[10]  J. Friedman,et al.  A Statistical View of Some Chemometrics Regression Tools , 1993 .

[11]  Neil A. Butler,et al.  The peculiar shrinkage properties of partial least squares regression , 2000 .

[12]  J. Whittaker.  Model Interpretation from the Additive Elements of the Likelihood Function , 1984 .

[13]  Paul H. C. Eilers,et al.  Generalized linear regression on sampled signals and curves: a P -spline approach , 1999 .

[14]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[15]  S. D. Jong.  SIMPLS: an alternative approach to partial least squares regression , 1993 .

[16]  A. E. Hoerl,et al.  Ridge regression: biased estimation for nonorthogonal problems , 2000 .