DABS: A Domain-Agnostic Benchmark for Self-Supervised Learning

Self-supervised learning algorithms, including BERT and SimCLR, have enabled 1 significant strides in fields like natural language processing, computer vision, and 2 speech processing. However, the domain-specificity of these algorithms means that 3 solutions must be handcrafted for each new setting, including myriad healthcare, 4 scientific, and multimodal domains. To catalyze progress towards more domain5 agnostic methods, we introduce DABS: a Domain-Agnostic Benchmark for Self6 supervised learning. To perform well on DABS, an algorithm must be pretrained 7 on six unlabeled datasets from diverse domains: natural images, text, speech 8 recordings, medical imaging, multichannel sensor data, and paired text and images, 9 and then perform well on a set of labeled tasks in each domain. We also present 10 e-Mix and ShED: two baseline domain-agnostic algorithms; their relatively modest 11 performance demonstrates that significant progress is needed before self-supervised 12 learning is an out-of-the-box solution for arbitrary domains. Code for benchmark 13 datasets and baseline algorithms is available at [redacted]. 14 Figure 1: The DABS Benchmark. A domain-agnostic self-supervised algorithm consists of 1) a model architecture, 2) an objective used to pretrain the model on unlabeled data, and 3) a transfer method used to deploy it on a downstream task (bolded items). A successful algorithm will achieve high performance on downstream tasks while holding these components constant across domains. †atamkin@stanford.edu Submitted to the 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks. Do not distribute.