Generalization Bounds for Domain Adaptation

In this paper, we provide a new framework to study the generalization bound of the learning process for domain adaptation. We consider two kinds of representative domain adaptation settings: one is domain adaptation with multiple sources and the other is domain adaptation combining source and target data. In particular, we use the integral probability metric to measure the difference between two domains. Then, we develop the specific Hoeffding-type deviation inequality and symmetrization inequality for either kind of domain adaptation to achieve the corresponding generalization bound based on the uniform entropy number. By using the resultant generalization bound, we analyze the asymptotic convergence and the rate of convergence of the learning process for domain adaptation. Meanwhile, we discuss the factors that affect the asymptotic behavior of the learning process. The numerical experiments support our results.

[1]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[2]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.

[3]  Jim Freeman.  Probability Metrics and the Stability of Stochastic Models , 1991 .

[4]  Jon A. Wellner,et al.  Weak Convergence and Empirical Processes: With Applications to Statistics , 1996 .

[5]  Vladimir Cherkassky,et al.  The Nature Of Statistical Learning Theory , 1997, IEEE Trans. Neural Networks.

[6]  A. Müller.  Integral Probability Metrics and Their Generating Classes of Functions , 1997, Advances in Applied Probability.

[7]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[8]  Vladimir Vapnik,et al.  An overview of statistical learning theory , 1999, IEEE Trans. Neural Networks.

[9]  Shahar Mendelson,et al.  A Few Notes on Statistical Learning Theory , 2002, Machine Learning Summer School.

[10]  Gábor Lugosi,et al.  Introduction to Statistical Learning Theory , 2004, Advanced Lectures on Machine Learning.

[11]  Thomas G. Dietterich,et al.  Improving SVM accuracy by training on auxiliary data sources , 2004, ICML.

[12]  P. Bartlett,et al.  Local Rademacher complexities , 2005, math/0508275.

[13]  Learning from Multiple Sources , 2006, Journal of Machine Learning Research.

[14]  Koby Crammer,et al.  Learning from Multiple Sources , 2006, NIPS.

[15]  Koby Crammer,et al.  Analysis of Representations for Domain Adaptation , 2006, NIPS.

[16]  John Blitzer,et al.  Domain Adaptation with Structural Correspondence Learning , 2006, EMNLP.

[17]  ChengXiang Zhai,et al.  Instance Weighting for Domain Adaptation in NLP , 2007, ACL.

[18]  John Blitzer,et al.  Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification , 2007, ACL.

[19]  Koby Crammer,et al.  Learning Bounds for Domain Adaptation , 2007, NIPS.

[20]  Steffen Bickel,et al.  Discriminative learning for differing training and test distributions , 2007, ICML '07.

[21]  Yishay Mansour,et al.  Domain Adaptation with Multiple Sources , 2008, NIPS.

[22]  Bharath K. Sriperumbudur,et al.  Ja n 20 09 A Note on Integral Probability Metrics and φ-divergences , 2009 .

[23]  Yishay Mansour,et al.  Multiple Source Adaptation and the Rényi Divergence , 2009, UAI.

[24]  Koby Crammer,et al.  A theory of learning from different domains , 2010, Machine Learning.

[25]  Yishay Mansour,et al.  Domain Adaptation: Learning Bounds and Algorithms , 2009, COLT.

[26]  Bernhard Schölkopf,et al.  A note on integral probability metrics and $\phi$-divergences , 2009, ArXiv.

[27]  John Shawe-Taylor,et al.  Improved Loss Bounds For Multiple Kernel Learning , 2011, AISTATS.

[28]  Mark D. Reid,et al.  Information, Divergence and Risk for Binary Experiments , 2009, J. Mach. Learn. Res..

[29]  Gert R. G. Lanckriet,et al.  On the empirical estimation of integral probability metrics , 2012 .

[30]  Dacheng Tao,et al.  This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART B: CYBERNETICS 1 Cross-Domain Human Action Recognition , 2022 .

[31]  Andreas Krause,et al.  Advances in Neural Information Processing Systems (NIPS) , 2014 .