BOOSTED UNSUPERVISED MULTI-SOURCE SELECTION FOR DOMAIN ADAPTATION

Abstract. Supervised machine learning needs high quality, densely sampled and labelled training data. Transfer learning (TL) techniques have been devised to reduce this dependency by adapting classifiers trained on different, but related, (source) training data to new (target) data sets. A problem in TL is how to quantify the relatedness of a source quickly and robustly, because transferring knowledge from unrelated data can degrade the performance of a classifier. In this paper, we propose a method that can select a nearly optimal source from a large number of candidate sources. This operation depends only on the marginal probability distributions of the data, thus allowing the use of the often abundant unlabelled data. We extend this method to multi-source selection by optimizing a weighted combination of sources. The source weights are computed using a very fast boosting-like optimization scheme. The run-time complexity of our method scales linearly in regard to the number of candidate sources and the size of the training set and is thus applicable to very large data sets. We also propose a modification of an existing TL algorithm to handle multiple weighted training sets. Our method is evaluated on five survey regions. The experiments show that our source selection method is effective in discriminating between related and unrelated sources, almost always generating results within 3% in overall accuracy of a classifier based on fully labelled training data. We also show that using the selected source as training data for a TL method will additionally result in a performance improvement.

[1]  Lorenzo Bruzzone,et al.  Toward the Automatic Updating of Land-Cover Maps by a Domain-Adaptation SVM Classifier and a Circular Validation Strategy , 2009, IEEE Transactions on Geoscience and Remote Sensing.

[2]  Sebastian Thrun,et al.  Learning to Learn: Introduction and Overview , 1998, Learning to Learn.

[3]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[4]  Klaus-Robert Müller,et al.  Covariate Shift Adaptation by Importance Weighted Cross Validation , 2007, J. Mach. Learn. Res..

[5]  Luc Van Gool,et al.  Iterative Nearest Neighbors for classification and dimensionality reduction , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Michael Cramer,et al.  The DGPF-Test on Digital Airborne Camera Evaluation - Over- view and Test Design , 2010 .

[7]  Luis Gómez-Chova,et al.  Graph Matching for Adaptation in Remote Sensing , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[8]  Gert R. G. Lanckriet,et al.  On the empirical estimation of integral probability metrics , 2012 .

[9]  Yoram Singer,et al.  Improved Boosting Algorithms Using Confidence-rated Predictions , 1998, COLT' 98.

[10]  Christian Heipke,et al.  ITERATIVE RE-WEIGHTED INSTANCE TRANSFER FOR DOMAIN ADAPTATION , 2016 .

[11]  Lorenzo Bruzzone,et al.  Domain Adaptation Problems: A DASVM Classification Technique and a Circular Validation Strategy , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Joydeep Ghosh,et al.  Transfer Learning with Cluster Ensembles , 2011, ICML Unsupervised and Transfer Learning.

[13]  Koby Crammer,et al.  Analysis of Representations for Domain Adaptation , 2006, NIPS.

[14]  Massih-Reza Amini,et al.  Semi Supervised Logistic Regression , 2002, ECAI.

[15]  Eric Eaton,et al.  Modeling Transfer Relationships Between Learning Tasks for Improved Inductive Transfer , 2008, ECML/PKDD.

[16]  Ming-Wei Chang,et al.  Analysis of switching dynamics with competing support vector machines , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[17]  Yuri K Shestopaloff.  Sums of exponential functions and their new fundamental properties , 2008 .

[18]  W. Press,et al.  Numerical Recipes: The Art of Scientific Computing , 1987 .

[19]  Li Cheng,et al.  Semi-supervised Domain Adaptation on Manifolds , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[20]  Yuhong Zhang,et al.  Logistic Regression for Transductive Transfer Learning from Multiple Sources , 2010, ADMA.

[21]  Bernhard Schölkopf,et al.  A Kernel Two-Sample Test , 2012, J. Mach. Learn. Res..

[22]  Bernice W. Polemis.  Nonparametric Statistics for the Behavioral Sciences , 1959 .

[23]  Lorenzo Bruzzone,et al.  Semisupervised Transfer Component Analysis for Domain Adaptation in Remote Sensing Image Classification , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[24]  Bianca Zadrozny,et al.  Learning and evaluating classifiers under sample selection bias , 2004, ICML.

[25]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[26]  Rama Chellappa,et al.  Domain adaptation for object recognition: An unsupervised approach , 2011, 2011 International Conference on Computer Vision.

[27]  Wojciech Zaremba,et al.  B-test: A Non-parametric, Low Variance Kernel Two-sample Test , 2013, NIPS.

[28]  Francesca Bovolo,et al.  A Novel Graph-Matching-Based Approach for Domain Adaptation in Classification of Remote Sensing Image Pair , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[29]  Surya S. Durbha,et al.  Evaluating transfer learning approaches for image information mining applications , 2011, 2011 IEEE International Geoscience and Remote Sensing Symposium.

[30]  Mark W. Schmidt,et al.  Accelerated training of conditional random fields with stochastic gradient methods , 2006, ICML.