Encoding sparse and competitive structures among tasks in multi-task learning

Abstract Multi-task learning (MTL) aims to enhance generalization performance by exploring the inherent structures across tasks. Most existing MTL methods are based on the assumption that the tasks are positively correlated, and utilize the shared structures among tasks to improve learning performance. By contrast, there also exist competitive structure (negative relationships) among tasks in some real-world applications, and conventional MTL methods which explore shared structures across tasks may lead to unsatisfactory performance in this setting. Another challenge, especially in a high dimensional setting, is to exclude irrelevant features (sparse structure) from the final model. For this purpose, this work propose a new method, which is referred to as Sparse Exclusive Lasso (SpEL) for multi-task learning. The proposed SpEL is able to capture the competitive relationship among tasks (competitive structure), while remove unimportant features which are common across the tasks from the final model (sparse structure). Experimental studies on synthetic and real data indicate that the proposed method can significantly improve learning performance by identifying sparse and task-competitive structures simultaneously.

[1]  Dit-Yan Yeung,et al.  Multi-task warped Gaussian process for personalized age estimation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[2]  Rich Caruana,et al.  Multitask Learning , 1997, Machine Learning.

[3]  Xiangyang Xue,et al.  Multiple Task Learning Using Iteratively Reweighted Least Square , 2013, IJCAI.

[4]  B. Torrésani,et al.  Structured Sparsity: from Mixed Norms to Structured Shrinkage , 2009 .

[5]  T. Golub,et al.  Gene expression-based classification of malignant gliomas correlates better with survival than histological classification. , 2003, Cancer research.

[6]  Massimiliano Pontil,et al.  Exploiting Unrelated Tasks in Multi-Task Learning , 2012, AISTATS.

[7]  Jieping Ye,et al.  Multi-Task Feature Learning Via Efficient l2, 1-Norm Minimization , 2009, UAI.

[8]  Jiayu Zhou,et al.  A multi-task learning formulation for predicting disease progression , 2011, KDD.

[9]  Daniel Hernández-Lobato,et al.  A Probabilistic Model for Dirty Multi-task Feature Selection , 2015, ICML.

[10]  Leon Wenliang Zhong,et al.  Convex Multitask Learning with Flexible Task Clusters , 2012, ICML.

[11]  Christoph H. Lampert,et al.  Learning to detect unseen object classes by between-class attribute transfer , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Hal Daumé,et al.  Learning Task Grouping and Overlap in Multi-task Learning , 2012, ICML.

[13]  Kilian Q. Weinberger,et al.  Large Margin Multi-Task Metric Learning , 2010, NIPS.

[14]  E. Lander,et al.  Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma subclasses , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[15]  Mohan S. Kankanhalli,et al.  Hierarchical Clustering Multi-Task Learning for Joint Human Action Grouping and Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Yudong D. He,et al.  Gene expression profiling predicts clinical outcome of breast cancer , 2002, Nature.

[17]  Eunho Yang,et al.  Asymmetric multi-task learning based on task relatedness and loss , 2016, ICML 2016.

[18]  Ali Jalali,et al.  A Dirty Model for Multi-task Learning , 2010, NIPS.

[19]  Massimiliano Pontil,et al.  Multi-Task Feature Learning , 2006, NIPS.

[20]  Jean-Philippe Vert,et al.  Clustered Multi-Task Learning: A Convex Formulation , 2008, NIPS.

[21]  Dit-Yan Yeung,et al.  A Convex Formulation for Learning Task Relationships in Multi-Task Learning , 2010, UAI.

[22]  Jieping Ye,et al.  Robust multi-task feature learning , 2012, KDD.

[23]  Bruno Torrésani,et al.  Sparsity and persistence: mixed norms provide simple signal models with dependent coefficients , 2009, Signal Image Video Process..

[24]  Lei Han,et al.  Learning Tree Structure in Multi-Task Learning , 2015, KDD.

[25]  Massimiliano Pontil,et al.  Regularized multi--task learning , 2004, KDD.

[26]  Stan Matwin,et al.  Machine Learning for the Detection of Oil Spills in Satellite Radar Images , 1998, Machine Learning.

[27]  Sinno Jialin Pan,et al.  Adaptive Group Sparse Multi-task Learning via Trace Lasso , 2017, IJCAI.

[28]  Fernando José Von Zuben,et al.  Spatial Projection of Multiple Climate Variables Using Hierarchical Multitask Learning , 2017, AAAI.

[29]  Dacheng Tao,et al.  Multi-task proximal support vector machine , 2015, Pattern Recognit..

[30]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[31]  Alexander J. Smola,et al.  Multitask Learning without Label Correspondences , 2010, NIPS.

[32]  Jianping Fan,et al.  Hierarchical learning of multi-task sparse metrics for large-scale image classification , 2017, Pattern Recognit..

[33]  J. Downing,et al.  Classification, subtype discovery, and prediction of outcome in pediatric acute lymphoblastic leukemia by gene expression profiling. , 2002, Cancer cell.

[34]  Rong Jin,et al.  Exclusive Lasso for Multi-task Feature Selection , 2010, AISTATS.

[35]  Aurelie C. Lozano,et al.  Multi-level Lasso for Sparse Multi-task Regression , 2012, ICML.

[36]  Jill P. Mesirov,et al.  Subclass Mapping: Identifying Common Subtypes in Independent Disease Data Sets , 2007, PloS one.

[37]  Lei Han,et al.  Learning Multi-Level Task Groups in Multi-Task Learning , 2015, AAAI.

[38]  Jiayu Zhou,et al.  Integrating low-rank and group-sparse structures for robust multi-task learning , 2011, KDD.

[39]  Jinbo Bi,et al.  Probabilistic Joint Feature Selection for Multi-task Learning , 2007, SDM.

[40]  Kristen Grauman,et al.  Learning with Whom to Share in Multi-task Feature Learning , 2011, ICML.

[41]  Qiang Zhou,et al.  Flexible Clustered Multi-Task Learning by Learning Representative Tasks , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.