USB: A Unified Semi-supervised Learning Benchmark

Semi-supervised learning (SSL) improves model generalization by leveraging massive unlabeled data to augment limited labeled samples. However, currently, popular SSL evaluation protocols are often constrained to computer vision (CV) tasks. In addition, previous work typically trains deep neural networks from scratch, which is time-consuming and environmentally unfriendly. To address the above issues, we construct a Unified SSL Benchmark (USB) by selecting 15 diverse, challenging, and comprehensive tasks from CV, natural language processing (NLP), and audio processing (Audio), on which we systematically evaluate dominant SSL methods, and also open-source a modular and extensible codebase for fair evaluation on these SSL methods. We further provide pre-trained versions of the state-of-the-art neural models for CV tasks to make the cost affordable for further tuning. USB enables the evaluation of a single SSL algorithm on more tasks from multiple domains but with less cost. Specifically, on a single NVIDIA V100, only 37 GPU days are required to evaluate FixMatch on 15 tasks in USB while 335 GPU days (279 GPU days on 4 CV datasets except for ImageNet) are needed on 5 CV tasks with the typical protocol.

[1]  Naoaki Okazaki,et al.  Semi-Supervised Formality Style Transfer with Consistency Training , 2022, ACL.

[2]  A. K. Qin,et al.  Self-Paced Co-Training of Graph Neural Networks for Semi-Supervised Node Classification , 2022, IEEE Transactions on Neural Networks and Learning Systems.

[3]  Fei Wang,et al.  SimMatch: Semi-supervised Learning with Similarity Matching , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Bernt Schiele,et al.  Revisiting Consistency Regularization for Semi-Supervised Learning , 2021, International Journal of Computer Vision.

[5]  B. Schiele,et al.  CoSSL: Co-Learning of Representation and Classifier for Imbalanced Semi-Supervised Learning , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Wanli Ouyang,et al.  Revisiting the Transferability of Supervised Pretraining: an MLP Perspective , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Ross B. Girshick,et al.  Masked Autoencoders Are Scalable Vision Learners , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Bingbing Ni,et al.  MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification , 2021, Scientific Data.

[9]  T. Shinozaki,et al.  FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling , 2021, NeurIPS.

[10]  Rong Jin,et al.  Dash: Semi-Supervised Learning with Dynamic Thresholding , 2021, ICML.

[11]  Xiang Bai,et al.  End-to-End Semi-Supervised Object Detection with Soft Teacher , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[12]  Ruslan Salakhutdinov,et al.  HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units , 2021, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[13]  Nicholas Carlini,et al.  AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation , 2021, ICLR.

[14]  Yihe Tang,et al.  Humble Teachers Teach Better Students for Semi-Supervised Object Detection , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Kate Saenko,et al.  OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers , 2021, NeurIPS.

[16]  Andy T. Liu,et al.  SUPERB: Speech processing Universal PERformance Benchmark , 2021, Interspeech.

[17]  Subhransu Maji,et al.  The Semi-Supervised iNaturalist-Aves Challenge at FGVC7 Workshop , 2021, ArXiv.

[18]  Zenglin Xu,et al.  A Survey on Deep Semi-Supervised Learning , 2021, IEEE Transactions on Knowledge and Data Engineering.

[19]  Peter Vajda,et al.  Unbiased Teacher for Semi-Supervised Object Detection , 2021, ICLR.

[20]  D. Tao,et al.  A Survey on Vision Transformer , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Matthieu Cord,et al.  Training data-efficient image transformers & distillation through attention , 2020, ICML.

[22]  Junnan Li,et al.  CoMatch: Semi-supervised Learning with Contrastive Graph Regularization , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[23]  Bingbing Ni,et al.  MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis , 2020, 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI).

[24]  Shu Hu,et al.  Uncertainty Aware Semi-Supervised Learning on Graph Data , 2020, NeurIPS.

[25]  S. Gelly,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2020, ICLR.

[26]  Juho Kannala,et al.  GraphMix: Improved Training of GNNs for Semi-Supervised Learning , 2020, AAAI.

[27]  Go Irie,et al.  Multi-Task Curriculum Framework for Open-Set Semi-Supervised Learning , 2020, ECCV.

[28]  Sung Ju Hwang,et al.  Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning , 2020, NeurIPS.

[29]  Zhi-Hua Zhou,et al.  Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data , 2020, ICML.

[30]  Xinyue Liu,et al.  SeqVAT: Virtual Adversarial Training for Semi-Supervised Sequence Labeling , 2020, ACL.

[31]  Abdel-rahman Mohamed,et al.  wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations , 2020, NeurIPS.

[32]  Geoffrey E. Hinton,et al.  Big Self-Supervised Models are Strong Semi-Supervised Learners , 2020, NeurIPS.

[33]  Zhi Xu,et al.  Rethinking the Value of Labels for Improving Class-Imbalanced Learning , 2020, NeurIPS.

[34]  C. Hudelot,et al.  An Overview of Deep Semi-Supervised Learning , 2020, ArXiv.

[35]  Qian Xu,et al.  Graph Random Neural Networks for Semi-Supervised Learning on Graphs , 2020, NeurIPS.

[36]  Yu Zhang,et al.  Conformer: Convolution-augmented Transformer for Speech Recognition , 2020, INTERSPEECH.

[37]  Diyi Yang,et al.  MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification , 2020, ACL.

[38]  Quoc V. Le,et al.  Meta Pseudo Labels , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Junnan Li,et al.  DivideMix: Learning with Noisy Labels as Semi-supervised Learning , 2020, ICLR.

[40]  Nojun Kwak,et al.  Class-Imbalanced Semi-Supervised Learning , 2020, ArXiv.

[41]  David Berthelot,et al.  FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence , 2020, NeurIPS.

[42]  Caiming Xiong,et al.  Proposal Learning for Semi-Supervised Object Detection , 2020, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[43]  Nicholas Carlini,et al.  ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring , 2019, ArXiv.

[44]  Jesper E. van Engelen,et al.  A survey on semi-supervised learning , 2019, Machine Learning.

[45]  André Susano Pinto,et al.  A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark , 2019, 1910.04867.

[46]  Quoc V. Le,et al.  Randaugment: Practical automated data augmentation with a reduced search space , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[47]  Marc'Aurelio Ranzato,et al.  Revisiting Self-Training for Neural Sequence Generation , 2019, ICLR.

[48]  Kevin Chen-Chuan Chang,et al.  Meta-GNN: Metagraph Neural Network for Semi-supervised learning in Attributed Heterogeneous Information Networks , 2019, 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[49]  Seong Joon Oh,et al.  CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[50]  Alexander Kolesnikov,et al.  S4L: Self-Supervised Semi-Supervised Learning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[51]  David Berthelot,et al.  MixMatch: A Holistic Approach to Semi-Supervised Learning , 2019, NeurIPS.

[52]  Ramón Fernández Astudillo,et al.  Self-supervised Sequence-to-sequence ASR using Unpaired Speech and Text , 2019, INTERSPEECH.

[53]  Quoc V. Le,et al.  Unsupervised Data Augmentation for Consistency Training , 2019, NeurIPS.

[54]  Jiebo Luo,et al.  Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[55]  Daniel P. W. Ellis,et al.  Learning Sound Event Classifiers from Web Audio with Noisy Labels , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[56]  Ramakant Nevatia,et al.  NOTE-RCNN: NOise Tolerant Ensemble RCNN for Semi-Supervised Object Detection , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[57]  Quoc V. Le,et al.  Semi-Supervised Sequence Modeling with Cross-View Training , 2018, EMNLP.

[58]  Georgios Kostopoulos,et al.  Semi-supervised regression: A recent review , 2018, J. Intell. Fuzzy Syst..

[59]  Andreas Dengel,et al.  Introducing Eurosat: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[60]  Stefano Ermon,et al.  Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance , 2018, NeurIPS.

[61]  Omer Levy,et al.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding , 2018, BlackboxNLP@EMNLP.

[62]  Colin Raffel,et al.  Realistic Evaluation of Deep Semi-Supervised Learning Algorithms , 2018, NeurIPS.

[63]  B Eswara Reddy,et al.  Semi-supervised learning: a brief review , 2018 .

[64]  Hongyi Zhang,et al.  mixup: Beyond Empirical Risk Minimization , 2017, ICLR.

[65]  Andreas Dengel,et al.  EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[66]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[67]  Abhishek Kumar,et al.  Semi-supervised Learning with GANs: Manifold Invariance with Improved Inference , 2017, NIPS.

[68]  Fan Yang,et al.  Good Semi-supervised Learning That Requires a Bad GAN , 2017, NIPS.

[69]  Shin Ishii,et al.  Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[70]  Harri Valpola,et al.  Weight-averaged consistency targets improve semi-supervised deep learning results , 2017, ArXiv.

[71]  Zhi-Hua Zhou,et al.  Learning Safe Prediction for Semi-Supervised Regression , 2017, AAAI.

[72]  Rob Fergus,et al.  Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks , 2016, ArXiv.

[73]  Yuxing Tang,et al.  Large Scale Semi-Supervised Object Detection Using Visual and Semantic Knowledge Transfer , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[74]  Augustus Odena,et al.  Semi-Supervised Learning with Generative Adversarial Networks , 2016, ArXiv.

[75]  Nikos Komodakis,et al.  Wide Residual Networks , 2016, BMVC.

[76]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[77]  Jost Tobias Springenberg,et al.  Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks , 2015, ICLR.

[78]  Quoc V. Le,et al.  Semi-supervised Sequence Learning , 2015, NIPS.

[79]  Karol J. Piczak.  ESC: Dataset for Environmental Sound Classification , 2015, ACM Multimedia.

[80]  Xiang Zhang,et al.  Character-level Convolutional Networks for Text Classification , 2015, NIPS.

[81]  Tapani Raiko,et al.  Semi-supervised Learning with Ladder Networks , 2015, NIPS.

[82]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[83]  Justin Salamon,et al.  A Dataset and Taxonomy for Urban Sound Research , 2014, ACM Multimedia.

[84]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[85]  Jure Leskovec,et al.  Hidden factors and hidden topics: understanding rating dimensions with review text , 2013, RecSys.

[86]  Eric Bair,et al.  Semi‐supervised clustering methods , 2013, Wiley interdisciplinary reviews. Computational statistics.

[87]  Honglak Lee,et al.  An Analysis of Single-Layer Networks in Unsupervised Feature Learning , 2011, AISTATS.

[88]  Guodong Zhou,et al.  Semi-Supervised Learning for Imbalanced Sentiment Classification , 2011, IJCAI.

[89]  Zhi-Hua Zhou,et al.  Towards Making Unlabeled Data Never Hurt , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[90]  Christopher Potts,et al.  Learning Word Vectors for Sentiment Analysis , 2011, ACL.

[91]  Xiaojin Zhu,et al.  Introduction to Semi-Supervised Learning , 2009, Synthesis Lectures on Artificial Intelligence and Machine Learning.

[92]  Ming-Wei Chang,et al.  Importance of Semantic Representation: Dataless Classification , 2008, AAAI.

[93]  Larry A. Wasserman,et al.  Statistical Analysis of Semi-Supervised Regression , 2007, NIPS.

[94]  William Stafford Noble,et al.  Support vector machine , 2013 .

[95]  Zhi-Hua Zhou,et al.  Semi-Supervised Regression with Co-Training , 2005, IJCAI.

[96]  Wei Li,et al.  Semi-Supervised Sequence Modeling with Syntactic Topic Models , 2005, AAAI.

[97]  Arindam Banerjee,et al.  Semi-supervised Clustering by Seeding , 2002, ICML.

[98]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[99]  M. Friedman.  A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings , 1940 .

[100]  M. Friedman.  The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance , 1937 .

[101]  Jihong Ouyang,et al.  Semi-Supervised Text Classification with Balanced Deep Representation Distributions , 2021, ACL.

[102]  Jian Yang,et al.  Universal Semi-Supervised Learning , 2021, NeurIPS.

[103]  Diyi Yang,et al.  Simple Conversational Data Augmentation for Semi-supervised Abstractive Dialogue Summarization , 2021, Conference on Empirical Methods in Natural Language Processing.

[104]  Feiyue Huang,et al.  On The Consistency Training for Open-Set Semi-Supervised Learning , 2021, ArXiv.

[105]  In So Kweon,et al.  Distribution-Aware Semantics-Oriented Pseudo-label for Imbalanced Semi-Supervised Learning , 2021, ArXiv.

[106]  Jeff A. Bilmes,et al.  Time-Consistent Self-Supervision for Semi-Supervised Learning , 2020, ICML.

[107]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[108]  Dong-Hyun Lee,et al.  Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks , 2013 .

[109]  Brendan T. O'Connor,et al.  Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics , 2011 .

[110]  Andrew Y. Ng,et al.  Reading Digits in Natural Images with Unsupervised Feature Learning , 2011 .

[111]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[112]  Nizar Grira,et al.  Unsupervised and Semi-supervised Clustering : a Brief Survey ∗ , 2004 .

[113]  John W. Merrill,et al.  Automatic Speech Recognition , 2005 .