Learning with dual heterogeneity: a nonparametric bayes model

Traditional data mining techniques are designed to model a single type of heterogeneity, such as multi-task learning for modeling task heterogeneity, multi-view learning for modeling view heterogeneity, etc. Recently, a variety of real applications emerged, which exhibit dual heterogeneity, namely both task heterogeneity and view heterogeneity. Examples include insider threat detection across multiple organizations, web image classification in different domains, etc. Existing methods for addressing such problems typically assume that multiple tasks are equally related and multiple views are equally consistent, which limits their application in complex settings with varying task relatedness and view consistency. In this paper, we advance state-of-the-art techniques by adaptively modeling task relatedness and view consistency via a nonparametric Bayes model: we model task relatedness using normal penalty with sparse covariances, and view consistency using matrix Dirichlet process. Based on this model, we propose the NOBLE algorithm using an efficient Gibbs sampler. Experimental results on multiple real data sets demonstrate the effectiveness of the proposed algorithm.

[1]  Craig A. Knoblock,et al.  Active + Semi-supervised Learning = Robust Multi-View Learning , 2002, ICML.

[2]  Trevor Darrell,et al.  Multi-View Learning in the Presence of View Disagreement , 2008, UAI 2008.

[3]  C. Yau,et al.  Bayesian Nonparametric Hidden Markov Models with application to the analysis of copy-number-variation in mammalian genomes. , 2011, Journal of the Royal Statistical Society. Series B, Statistical methodology.

[4]  Dan Zhang,et al.  Multi-view transfer learning with a large margin approach , 2011, KDD.

[5]  D. Blackwell,et al.  Ferguson Distributions Via Polya Urn Schemes , 1973 .

[6]  Jiayu Zhou,et al.  Clustered Multi-Task Learning Via Alternating Structure Optimization , 2011, NIPS.

[7]  Rayid Ghani,et al.  Analyzing the effectiveness and applicability of co-training , 2000, CIKM '00.

[8]  Sebastian Thrun,et al.  Discovering Structure in Multiple Learning Tasks: The TC Algorithm , 1996, ICML.

[9]  C. Antoniak.  Mixtures of Dirichlet Processes with Applications to Bayesian Nonparametric Problems , 1974 .

[10]  Shannon L. Risacher,et al.  Identifying disease sensitive and quantitative trait-relevant biomarkers from multidimensional heterogeneous imaging genetics data via sparse multimodal multitask learning , 2012, Bioinform..

[11]  Jieping Ye,et al.  A convex formulation for learning shared structures from multiple tasks , 2009, ICML '09.

[12]  Volker Tresp,et al.  A nonparametric hierarchical bayesian framework for information filtering , 2004, SIGIR '04.

[13]  Lawrence Carin,et al.  Cross-Domain Multitask Learning with Latent Probit Models , 2012, ICML.

[14]  Sham M. Kakade,et al.  Multi-view Regression Via Canonical Correlation Analysis , 2007, COLT.

[15]  Lawrence Carin,et al.  Semi-Supervised Multitask Learning , 2007, NIPS.

[16]  Jiayu Zhou,et al.  Integrating low-rank and group-sparse structures for robust multi-task learning , 2011, KDD.

[17]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[18]  Jingrui He,et al.  A Graphbased Framework for Multi-Task Multi-View Learning , 2011, ICML.

[19]  Onno Zoeter,et al.  Sparse Bayesian Multi-Task Learning , 2011, NIPS.

[20]  D. Burr,et al.  A Bayesian Semiparametric Model for Random-Effects Meta-Analysis , 2005 .

[21]  Jieping Ye,et al.  Drosophila Gene Expression Pattern Annotation through Multi-Instance Multi-Label Learning , 2009, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[22]  Maayan Harel,et al.  Learning from Multiple Outlooks , 2010, ICML.

[23]  Zhi-Hua Zhou,et al.  A New Analysis of Co-Training , 2010, ICML.

[24]  Thomas L. Griffiths,et al.  Hierarchical Topic Models and the Nested Chinese Restaurant Process , 2003, NIPS.

[25]  Dan Zhang,et al.  MI2LS: multi-instance learning from multiple informationsources , 2013, KDD.

[26]  Wei Chu,et al.  Gaussian Process Models for Link Analysis and Transfer Learning , 2007, NIPS.

[27]  John Shawe-Taylor,et al.  Two view learning: SVM-2K, Theory and Practice , 2005, NIPS.

[28]  T. Ferguson.  A Bayesian Analysis of Some Nonparametric Problems , 1973 .

[29]  Lawrence Carin,et al.  Multi-Task Learning for Classification with Dirichlet Process Priors , 2007, J. Mach. Learn. Res..

[30]  Rong Yan,et al.  Interactive Image Segmentation Using Dirichlet Process Multiple-View Learning , 2012, IEEE Transactions on Image Processing.

[31]  Tong Zhang,et al.  A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , 2005, J. Mach. Learn. Res..

[32]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[33]  L. Carin,et al.  The Matrix Stick-Breaking Process , 2008 .

[34]  Wei-Ying Ma,et al.  Collaborative Ensemble Learning: Combining Collaborative and Content-Based Information Filtering via Hierarchical Bayes , 2002, UAI.

[35]  Fuzhen Zhuang,et al.  Shared Structure Learning for Multiple Tasks with Multiple Views , 2013, ECML/PKDD.

[36]  Eric O. Postma,et al.  Dimensionality Reduction: A Comparative Review , 2008 .

[37]  Tom Heskes,et al.  Task Clustering and Gating for Bayesian Multitask Learning , 2003, J. Mach. Learn. Res..

[38]  Jieping Ye,et al.  Learning Incoherent Sparse and Low-Rank Patterns from Multiple Tasks , 2010, TKDD.

[39]  Hal Daumé,et al.  Co-regularized Multi-view Spectral Clustering , 2011, NIPS.

[40]  Jintao Zhang,et al.  Inductive multi-task learning with multiple view data , 2012, KDD.

[41]  Dit-Yan Yeung,et al.  A Convex Formulation for Learning Task Relationships in Multi-Task Learning , 2010, UAI.

[42]  Anton Schwaighofer,et al.  Learning Gaussian processes from multiple tasks , 2005, ICML.

[43]  Yan Liu,et al.  Linking Heterogeneous Input Spaces with Pivots for Multi-Task Learning , 2014, SDM.