NAS-Bench-x11 and the Power of Learning Curves

In the past few years, algorithms for neural architecture search (NAS) have been used to automatically find architectures that achieve state-of-the-art performance on various datasets. In 2019, there were calls for reproducible and fair comparisons within NAS research [15, 16] due to both the lack of a consistent training pipeline between papers and experiments with not enough trials to reach statistically significant conclusions. These concerns spurred the release of tabular benchmarks, such as NAS-Bench-101 [25] and NASBench-201 [4], created by fully training all non-isomorphic architectures in search spaces of size 423 000 and 6 466, respectively. These benchmarks allow researchers to easily simulate NAS experiments, making it possible to quickly run fair NAS comparisons and to run enough trials to reach statistical significance at very little computational cost [9]. Recently, to extend the benefits of tabular NAS benchmarks to larger, more realistic NAS search spaces which cannot be evaluated exhaustively, it was proposed to construct surrogate NAS benchmarks. The first such surrogate benchmark is NAS-Bench-301 [21], which was created by training 60 000 architectures from the DARTS [17] search space, and then fitting a surrogate model which can be used to estimate the performance of all 10 architectures in the DARTS search space. Since 2019, dozens of papers have used these NAS benchmarks to develop new algorithms. A downside of these benchmarks is that the main type of algorithms that can be benchmarked are single fidelity algorithms: when the NAS algorithm chooses to evaluate an architecture, the architecture is fully trained and only the final validation accuracy is outputted. This is because NAS-Bench-301 only contains the architectures‘ accuracy