Mixed-norm partial least squares

Abstract The partial least squares (PLS) method is designed for prediction problems when the number of predictors is larger than the number of training samples. PLS is based on latent components which are linear combinations of the original predictors, it automatically employs all predictors regardless of their relevance. This strategy will potentially degrade its performance, and make the obtained coefficients lack interpretability. Then, several sparse PLS (SPLS) methods are proposed to simultaneously conduct prediction and variable selection via sparsely combining original predictors. However, if information bleed across different components, common variables shared by these components should be selected with successive loadings. To address this issue, we propose a new SPLS model — mixed-norm PLS (MNPLS) — to select common variables during each deflation in this paper. More specifically, we introduced the l 2,1 norm to the direction matrix and then developed the corresponding solution to MNPLS. Moreover, we also conducted convergence analysis to mathematically support the proposed MNPLS. Experiments are conducted on four real datasets, experimental results verified our theoretical analysis, and also demonstrated that our MNPLS method can generally outperform the standard PLS and other existing methods in variable selection and prediction.

[1]  Johan Trygg,et al.  The PLS method -- partial least squares projections to latent structures -- and its applications in industrial RDP (research, development, and production) , 2004 .

[2]  Xinge You,et al.  Variance constrained partial least squares , 2015 .

[3]  J. Friedman,et al.  A Statistical View of Some Chemometrics Regression Tools , 1993 .

[4]  Larry S. Davis,et al.  Human detection using partial least squares analysis , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[5]  Søren Balling Engelsen,et al.  Depth profiling of porcine adipose tissue by Raman spectroscopy , 2012 .

[6]  Kristian Hovde Liland,et al.  Powered partial least squares discriminant analysis , 2009 .

[7]  S. Wold,et al.  The Collinearity Problem in Linear Regression. The Partial Least Squares (PLS) Approach to Generalized Inverses , 1984 .

[8]  Peter Bühlmann.  Regression shrinkage and selection via the Lasso: a retrospective (Robert Tibshirani): Comments on the presentation , 2011 .

[9]  I. Jolliffe,et al.  A Modified Principal Component Technique Based on the LASSO , 2003 .

[10]  S. Wold,et al.  Wavelength interval selection in multicomponent spectral analysis by moving window partial least-squares regression with applications to mid-infrared and near-infrared spectroscopic data. , 2002, Analytical chemistry.

[11]  C. Ruckebusch,et al.  Genetic algorithm optimisation combined with partial least squares regression and mutual information variable selection procedures in near-infrared quantitative analysis of cotton-viscose textiles. , 2007, Analytica chimica acta.

[12]  Feiping Nie,et al.  Efficient and Robust Feature Selection via Joint ℓ2, 1-Norms Minimization , 2010, NIPS.

[13]  Dong-Sheng Cao,et al.  Recipe for revealing informative metabolites based on model population analysis , 2010, Metabolomics.

[14]  B. Kowalski,et al.  Partial least-squares regression: a tutorial , 1986 .

[15]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[16]  S. Keleş,et al.  Sparse partial least squares regression for simultaneous dimension reduction and variable selection , 2010, Journal of the Royal Statistical Society. Series B, Statistical methodology.

[17]  John Hulland,et al.  Use of partial least squares (PLS) in strategic management research: a review of four recent studies , 1999 .

[18]  J. Westerhuis,et al.  Multivariate modelling of the pharmaceutical two‐step process of wet granulation and tableting with multiblock partial least squares , 1997 .

[19]  R D Bautista,et al.  Simultaneous spectrophotometric determination of drugs in pharmaceutical preparations using multiple linear regression and partial least-squares regression, calibration and prediction methods. , 1996, Talanta.

[20]  Ulf G. Indahl,et al.  A twist to partial least squares regression , 2005 .

[21]  S. Wold,et al.  PLS-regression: a basic tool of chemometrics , 2001 .

[22]  Youngjo Lee,et al.  Sparse partial least-squares regression and its applications to high-throughput data analysis , 2011 .

[23]  Erik Andries,et al.  Spectral Multivariate Calibration with Wavelength Selection Using Variants of Tikhonov Regularization , 2010, Applied spectroscopy.

[24]  W. F. McClure,et al.  Near Infrared Technology for Precision Environmental Measurements: Part 1. Determination of Nitrogen in Green- and Dry-Grass Tissue , 2002 .

[25]  Jianhua Z. Huang,et al.  Sparse principal component analysis via regularized low rank matrix approximation , 2008 .

[26]  Massimiliano Pontil,et al.  Multi-Task Feature Learning , 2006, NIPS.

[27]  M. Dyrby,et al.  Chemometric Quantitation of the Active Substance (Containing C≡N) in a Pharmaceutical Tablet Using Near-Infrared (NIR) Transmittance and NIR FT-Raman Spectra , 2002 .

[28]  E. Lughofer,et al.  Enhanced genetic operators design for waveband selection in multivariate calibration based on NIR spectroscopy , 2014 .

[29]  R. Tibshirani,et al.  Regression shrinkage and selection via the lasso: a retrospective , 2011 .

[30]  J. V. Haxby,et al.  Spatial Pattern Analysis of Functional Brain Images Using Partial Least Squares , 1996, NeuroImage.

[31]  Hongdong Li,et al.  Key wavelengths screening using competitive adaptive reweighted sampling method for multivariate calibration. , 2009, Analytica chimica acta.

[32]  Joe Whittaker,et al.  Adaptively preconditioned Krylov spaces to identify irrelevant predictors , 2010 .

[33]  A. Höskuldsson.  PLS regression methods , 1988 .

[34]  Zi Huang,et al.  Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence ℓ2,1-Norm Regularized Discriminative Feature Selection for Unsupervised Learning , 2022 .

[35]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[36]  Tahir Mehmood,et al.  A review of variable selection methods in Partial Least Squares Regression , 2012 .

[37]  Philippe Besse,et al.  Statistical Applications in Genetics and Molecular Biology A Sparse PLS for Variable Selection when Integrating Omics Data , 2011 .

[38]  D. Massart,et al.  Elimination of uninformative variables for multivariate calibration. , 1996, Analytical chemistry.