A Note on k-support Norm Regularized Risk Minimization

The k-support norm has been recently introduced to perform correlated sparsity regularization. Although Argyriou et al. only reported experiments using squared loss, here we apply it to several other commonly used settings resulting in novel machine learning algorithms with interesting and familiar limit cases. Source code for the algorithms described here is available.

[1]  A. Ng.  Feature selection, L1 vs. L2 regularization, and rotational invariance , 2004, Twenty-first international conference on Machine learning - ICML '04.

[2]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[3]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[4]  A Tikhonov,et al.  Solution of Incorrectly Formulated Problems and the Regularization Method , 1963 .

[5]  Michael D. Gordon,et al.  Regularized Least Absolute Deviations Regression and an Efficient Algorithm for Parameter Tuning , 2006, Sixth International Conference on Data Mining (ICDM'06).

[6]  Robert Tibshirani,et al.  1-norm Support Vector Machines , 2003, NIPS.

[7]  Matthew B. Blaschko,et al.  FMRI analysis of cocaine addiction using k-support sparsity , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[8]  Charles A. Micchelli,et al.  When is there a representer theorem? Vector versus matrix regularizers , 2008, J. Mach. Learn. Res..

[9]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[10]  Nathan Srebro,et al.  Sparse Prediction with the $k$-Support Norm , 2012, NIPS.

[11]  H. Zou,et al.  The doubly regularized support vector machine , 2006 .

[12]  Michael I. Jordan,et al.  Convexity, Classification, and Risk Bounds , 2006 .

[13]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[14]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[15]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[16]  Olivier Chapelle,et al.  Training a Support Vector Machine in the Primal , 2007, Neural Computation.

[17]  A. Atiya,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2005, IEEE Transactions on Neural Networks.