Domain Adaptation under Open Set Label Shift

We introduce the problem of domain adaptation under Open Set Label Shift (OSLS) where the label distribution can change arbitrarily and a new class may arrive during deployment, but the class-conditional distributions p p x | y q are domain-invariant. The learner’s goals here are two-fold: (a) estimate the target label distribution, includ-ing the novel class; and (b) learn a target classifier. First, we establish necessary and sufficient conditions for identifying these quantities. Second, we propose practical methods for both tasks. Unlike typical Open Set Domain Adaptation (OSDA) problems, which tend to be ill-posed and amenable only to heuristics, OSLS offers a well-posed problem amenable to more principled machinery. Experiments across numerous semi-synthetic benchmarks on vision, language, and medical datasets demonstrate that our methods consistently outperform OSDA baselines, achieving 10 – 25% improvements in target domain accuracy. Finally, we analyze the proposed methods, establishing finite-sample convergence to the true label marginal and convergence to an optimal classifier for linear models in a Gaussian setup. To mitigate the error accumulation problem, we propose the PULSE framework in the next section.

[1]  Zachary Chase Lipton,et al.  Leveraging Unlabeled Data to Predict Out-of-Distribution Performance , 2022, ICLR.

[2]  Tatsunori B. Hashimoto,et al.  Extending the WILDS Benchmark for Unsupervised Adaptation , 2021, ICLR.

[3]  Jure Leskovec,et al.  Open-World Semi-Supervised Learning , 2021, ICLR.

[4]  Sivaraman Balakrishnan,et al.  Mixture Proportion Estimation and PU Learning: A Modern Approach , 2021, NeurIPS.

[5]  Sivaraman Balakrishnan,et al.  RATT: Leveraging Unlabeled Data to Guarantee Generalization , 2021, ICML.

[6]  Pang Wei Koh,et al.  WILDS: A Benchmark of in-the-Wild Distribution Shifts , 2020, ICML.

[7]  Suvrit Sra,et al.  Coping with Label Shift via Distributionally Robust Optimisation , 2020, ICLR.

[8]  Aleksander Madry,et al.  BREEDS: Benchmarks for Subpopulation Shift , 2020, ICLR.

[9]  Jure Leskovec,et al.  Concept Learners for Few-Shot Learning , 2020, ICLR.

[10]  Dmitry Ivanov.  DEDPUL: Difference-of-Estimated-Densities-based Positive-Unlabeled Learning , 2019, 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA).

[11]  Tatiana Tommasi,et al.  On the Effectiveness of Image Rotation for Open Set Domain Adaptation , 2020, ECCV.

[12]  Sheng Liu,et al.  Early-Learning Regularization Prevents Memorization of Noisy Labels , 2020, NeurIPS.

[13]  Chen Gong,et al.  Self-PU: Self Boosted and Calibrated Positive-Unlabeled Training , 2020, ICML.

[14]  Ang Li,et al.  Hybrid Models for Open Set Recognition , 2020, ECCV.

[15]  Sivaraman Balakrishnan,et al.  A Unified View of Label Shift Estimation , 2020, NeurIPS.

[16]  Kate Saenko,et al.  Universal Domain Adaptation through Self Supervision , 2020, NeurIPS.

[17]  Geoffrey E. Hinton,et al.  A Simple Framework for Contrastive Learning of Visual Representations , 2020, ICML.

[18]  David Berthelot,et al.  FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence , 2020, NeurIPS.

[19]  Jesse Davis,et al.  Learning from positive and unlabeled data: a survey , 2018, Machine Learning.

[20]  Mingsheng Long,et al.  Learning to Detect Open Classes for Universal Domain Adaptation , 2020, ECCV.

[21]  Avanti Shrikumar,et al.  Adapting to Label Shift with Bias-Corrected Calibration , 2019 .

[22]  Angela Oliveira Pisco,et al.  A Single Cell Transcriptomic Atlas Characterizes Aging Tissues in the Mouse , 2019, bioRxiv.

[23]  Sebastian Nowozin,et al.  Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift , 2019, NeurIPS.

[24]  Michael I. Jordan,et al.  Universal Domain Adaptation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Hong Liu,et al.  Separate to Adapt: Open Set Domain Adaptation via Progressive Separation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Wen Li,et al.  Known-class Aware Self-ensemble for Open Set Domain Adaptation , 2019, ArXiv.

[27]  Wei-Shi Zheng,et al.  Weakly Supervised Open-Set Domain Adaptation by Dual-Domain Collaboration , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Yuchen Zhang,et al.  Bridging Theory and Algorithm for Domain Adaptation , 2019, ICML.

[29]  Jianmin Wang,et al.  Learning to Transfer Examples for Partial Domain Adaptation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Kamyar Azizzadenesheli,et al.  Regularized Learning for Domain Adaptation under Label Shifts , 2019, ICLR.

[31]  Ruosong Wang,et al.  Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks , 2019, ICML.

[32]  Bo Wang,et al.  Moment Matching for Multi-Source Domain Adaptation , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[33]  Vijayan K. Asari,et al.  Breast Cancer Classification from Histopathological Images with Inception Recurrent Residual Convolutional Neural Network , 2018, Journal of Digital Imaging.

[34]  Tom Drummond,et al.  Learning Factorized Representations for Open-set Domain Adaptation , 2018, ICLR.

[35]  Dong Xu,et al.  Collaborative and Adversarial Network for Unsupervised Domain Adaptation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[36]  Maya R. Gupta,et al.  To Trust Or Not To Trust A Classifier , 2018, NeurIPS.

[37]  Tatsuya Harada,et al.  Open Set Domain Adaptation by Backpropagation , 2018, ECCV.

[38]  Alexander J. Smola,et al.  Detecting and Correcting for Label Shift with Black Box Predictors , 2018, ICML.

[39]  Jesse Davis,et al.  Estimating the Class Prior in Positive and Unlabeled Data Through Decision Tree Induction , 2018, AAAI.

[40]  Nathan Srebro,et al.  The Implicit Bias of Gradient Descent on Separable Data , 2017, J. Mach. Learn. Res..

[41]  Juergen Gall,et al.  Open Set Domain Adaptation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[42]  Dacheng Tao,et al.  Multi-Positive and Unlabeled Learning , 2017, IJCAI.

[43]  Sethuraman Panchanathan,et al.  Deep Hashing Network for Unsupervised Domain Adaptation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Ran El-Yaniv,et al.  Selective Classification for Deep Neural Networks , 2017, NIPS.

[45]  Gang Niu,et al.  Positive-Unlabeled Learning with Non-Negative Risk Estimator , 2017, NIPS.

[46]  Kate Saenko,et al.  Correlation Alignment for Unsupervised Domain Adaptation , 2016, Domain Adaptation in Computer Vision Applications.

[47]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[48]  Samy Bengio,et al.  Understanding deep learning requires rethinking generalization , 2016, ICLR.

[49]  Kevin Gimpel,et al.  A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks , 2016, ICLR.

[50]  Michael I. Jordan,et al.  Deep Transfer Learning with Joint Adaptation Networks , 2016, ICML.

[51]  Kate Saenko,et al.  Deep CORAL: Correlation Alignment for Deep Domain Adaptation , 2016, ECCV Workshops.

[52]  Luiz Eduardo Soares de Oliveira,et al.  A Dataset for Breast Cancer Histopathological Image Classification , 2016, IEEE Transactions on Biomedical Engineering.

[53]  Ambuj Tewari,et al.  Mixture Proportion Estimation via Kernel Embeddings of Distributions , 2016, ICML.

[54]  Martha White,et al.  Nonparametric semi-supervised learning of class proportions , 2016, ArXiv.

[55]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  MarchandMario,et al.  Domain-adversarial training of neural networks , 2016 .

[57]  Gang Niu,et al.  Convex Formulation for Learning from Positive and Unlabeled Data , 2015, ICML.

[58]  Clayton Scott,et al.  A Rate of Convergence for Mixture Proportion Estimation, with Application to Learning from Noisy Labels , 2015, AISTATS.

[59]  Michael I. Jordan,et al.  Learning Transferable Features with Deep Adaptation Networks , 2015, ICML.

[60]  Terrance E. Boult,et al.  Towards Open World Recognition , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[62]  Haofu Liao,et al.  A Deep Learning Approach to Universal Skin Disease Classification , 2015 .

[63]  Gang Niu,et al.  Analysis of Learning from Positive and Unlabeled Data , 2014, NIPS.

[64]  Masashi Sugiyama,et al.  Class Prior Estimation from Positive and Unlabeled Data , 2014, IEICE Trans. Inf. Syst..

[65]  Clayton Scott,et al.  Class Proportion Estimation with Application to Multiclass Anomaly Rejection , 2013, AISTATS.

[66]  Mehryar Mohri,et al.  Domain adaptation and sample bias correction theory and algorithm for regression , 2014, Theor. Comput. Sci..

[67]  Anderson Rocha,et al.  Toward Open Set Recognition , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[68]  Bernhard Schölkopf,et al.  Domain Adaptation under Target and Conditional Shift , 2013, ICML.

[69]  Yishay Mansour,et al.  Learning Bounds for Importance Weighting , 2010, NIPS.

[70]  Trevor Darrell,et al.  Adapting Visual Category Models to New Domains , 2010, ECCV.

[71]  Tyler Lu,et al.  Impossibility Theorems for Domain Adaptation , 2010, AISTATS.

[72]  Gilles Blanchard,et al.  Semi-Supervised Novelty Detection , 2010, J. Mach. Learn. Res..

[73]  Karsten M. Borgwardt,et al.  Covariate Shift by Kernel Mean Matching , 2009, NIPS 2009.

[74]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[75]  Neil D. Lawrence,et al.  When Training and Test Sets Are Different: Characterizing Learning Transfer , 2009 .

[76]  Charles Elkan,et al.  Learning classifiers from only positive and unlabeled data , 2008, KDD.

[77]  Bianca Zadrozny,et al.  Learning and evaluating classifiers under sample selection bias , 2004, ICML.

[78]  Marco Saerens,et al.  Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure , 2002, Neural Computation.

[79]  Rémi Gilleron,et al.  Learning from positive and unlabeled examples , 2000, Theor. Comput. Sci..

[80]  Rémi Gilleron,et al.  Positive and Unlabeled Examples Help Learning , 1999, ALT.