Adaptation Regularization: A General Framework for Transfer Learning

Domain transfer learning, which learns a target classifier using labeled data from a different distribution, has shown promising value in knowledge discovery yet still been a challenging problem. Most previous works designed adaptive classifiers by exploring two learning strategies independently: distribution adaptation and label propagation. In this paper, we propose a novel transfer learning framework, referred to as Adaptation Regularization based Transfer Learning (ARTL), to model them in a unified way based on the structural risk minimization principle and the regularization theory. Specifically, ARTL learns the adaptive classifier by simultaneously optimizing the structural risk functional, the joint distribution matching between domains, and the manifold consistency underlying marginal distribution. Based on the framework, we propose two novel methods using Regularized Least Squares (RLS) and Support Vector Machines (SVMs), respectively, and use the Representer theorem in reproducing kernel Hilbert space to derive corresponding solutions. Comprehensive experiments verify that ARTL can significantly outperform state-of-the-art learning methods on several public text and image datasets.

[1]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[2]  Bernhard Schölkopf,et al.  A Generalized Representer Theorem , 2001, COLT/EuroCOLT.

[3]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[4]  Koby Crammer,et al.  Analysis of Representations for Domain Adaptation , 2006, NIPS.

[5]  Bernhard Schölkopf,et al.  A Kernel Method for the Two-Sample-Problem , 2006, NIPS.

[6]  John Blitzer,et al.  Domain Adaptation with Structural Correspondence Learning , 2006, EMNLP.

[7]  Massimiliano Pontil,et al.  Multi-Task Feature Learning , 2006, NIPS.

[8]  Qiang Yang,et al.  Co-clustering based classification for out-of-domain documents , 2007, KDD '07.

[9]  Rong Yan,et al.  Cross-domain video concept detection using adaptive svms , 2007, ACM Multimedia.

[10]  Qiang Yang,et al.  Boosting for transfer learning , 2007, ICML '07.

[11]  ChengXiang Zhai,et al.  Instance Weighting for Domain Adaptation in NLP , 2007, ACL.

[12]  Lawrence Carin,et al.  Semi-Supervised Multitask Learning , 2007, NIPS.

[13]  Jiawei Han,et al.  Spectral Regression: A Unified Approach for Sparse Subspace Learning , 2007, Seventh IEEE International Conference on Data Mining (ICDM 2007).

[14]  Qiang Yang,et al.  Transfer Learning via Dimensionality Reduction , 2008, AAAI.

[15]  Tong Zhang,et al.  Graph-Based Semi-Supervised Learning and Spectral Kernel Design , 2008, IEEE Transactions on Information Theory.

[16]  Jiawei Han,et al.  Knowledge transfer via multiple model local structure mapping , 2008, KDD.

[17]  Qiang Yang,et al.  Spectral domain-transfer learning , 2008, KDD.

[18]  Jun Huan,et al.  Large margin transductive transfer learning , 2009, CIKM.

[19]  Ivor W. Tsang,et al.  Extracting discriminative concepts for domain adaptation in text mining , 2009, KDD.

[20]  Deepak S. Turaga,et al.  Cross domain distribution adaptation via kernel mapping , 2009, KDD.

[21]  Qiang Yang,et al.  Transfer learning for collaborative filtering via a rating-matrix generative model , 2009, ICML '09.

[22]  Dacheng Tao,et al.  Bregman Divergence-Based Regularization for Transfer Subspace Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[23]  Bernt Schiele,et al.  What helps where – and why? Semantic relatedness for knowledge transfer , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[24]  Lorenzo Bruzzone,et al.  Domain Adaptation Problems: A DASVM Classification Technique and a Circular Validation Strategy , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Avishek Saha,et al.  Co-regularization Based Semi-supervised Domain Adaptation , 2010, NIPS.

[26]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[27]  Qiang Yang,et al.  Cross-domain sentiment classification via spectral feature alignment , 2010, WWW '10.

[28]  Ivor W. Tsang,et al.  Domain Adaptation via Transfer Component Analysis , 2009, IEEE Transactions on Neural Networks.

[29]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[30]  Yong Yu,et al.  Video summarization via transferrable structured learning , 2011, WWW.

[31]  Xiaojun Wu,et al.  Graph Regularized Nonnegative Matrix Factorization for Data Representation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Qiang Yang,et al.  Heterogeneous Transfer Learning for Image Classification , 2011, AAAI.

[33]  Yan Liu,et al.  Learning with Minimum Supervision: A General Framework for Transductive Transfer Learning , 2011, 2011 IEEE 11th International Conference on Data Mining.

[34]  Chang Wang,et al.  Heterogeneous Domain Adaptation Using Manifold Alignment , 2011, IJCAI.

[35]  XiongHui,et al.  Mining Distinction and Commonality across Multiple Domains Using Generative Model for Text Classification , 2012 .

[36]  Hui Xiong,et al.  Mining Distinction and Commonality across Multiple Domains Using Generative Model for Text Classification , 2012, IEEE Transactions on Knowledge and Data Engineering.

[37]  Ivor W. Tsang,et al.  Domain Transfer Multiple Kernel Learning , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Jun Huan,et al.  Knowledge Transfer with Low-Quality Data: A Feature Extraction Issue , 2011, IEEE Transactions on Knowledge and Data Engineering.

[39]  Jianmin Wang,et al.  Dual Transfer Learning , 2012, SDM.

[40]  Min Xiao,et al.  Semi-Supervised Kernel Matching for Domain Adaptation , 2012, AAAI.

[41]  Yuhong Xiong,et al.  Erratum to "Mining Distinction and Commonality across Multiple Domains Using Generative Model for Text Classification" , 2012, IEEE Trans. Knowl. Data Eng..

[42]  Korris Fu-Lai Chung,et al.  On minimum distribution discrepancy support vector machine for domain adaptation , 2012, Pattern Recognit..

[43]  Philip S. Yu,et al.  Transfer across Completely Different Feature Spaces via Spectral Embedding , 2013, IEEE Transactions on Knowledge and Data Engineering.

[44]  Jianmin Wang,et al.  Transfer Learning with Graph Co-Regularization , 2012, IEEE Transactions on Knowledge and Data Engineering.