Neural Architecture Search Benchmarks: Insights and Survey

Neural Architecture Search (NAS), a promising and fast-moving research field, aims to automate the architectural design of Deep Neural Networks (DNNs) to achieve better performance on the given task and dataset. NAS methods have been very successful in discovering efficient models for various Computer Vision, Natural Language Processing, etc. The major obstacles to the advancement of NAS techniques are the demand for large computation resources and fair evaluation of various search methods. The differences in training pipeline and setting make it challenging to compare the efficiency of two NAS algorithms. A large number of NAS Benchmarks to simulate the architecture evaluation in seconds have been released over the last few years to ease the computation burden of training neural networks and can aid in the unbiased assessment of different search methods. This paper provides an extensive review of several publicly available NAS Benchmarks in the literature. We provide technical details and a deeper understanding of each benchmark and point out future directions.

[1]  F. Hutter,et al.  Neural Architecture Search: Insights from 1000 Papers , 2023, ArXiv.

[2]  G. Yen,et al.  Efficient Evaluation Methods for Neural Architecture Search: A Survey , 2023, ArXiv.

[3]  Krishna Teja Chitty-Venkata,et al.  Neural Architecture Search Survey: A Hardware Perspective , 2022, ACM Comput. Surv..

[4]  Mengjie Zhang,et al.  A Survey on Evolutionary Neural Architecture Search , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[5]  Thomas C. P. Chau,et al.  BLOX: Macro Neural Architecture Search Benchmark and Algorithms , 2022, NeurIPS.

[6]  F. Hutter,et al.  NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies , 2022, NeurIPS.

[7]  M. Garland,et al.  Efficient Sparsely Activated Transformers , 2022, ArXiv.

[8]  Yi Qin,et al.  NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search , 2022, NeurIPS.

[9]  Yaochu Jin,et al.  A survey on computationally efficient neural architecture search , 2022, Journal of Automation and Intelligence.

[10]  Se-Young Yun,et al.  A Survey of Supernet Optimization and its Applications: Spatial and Temporal Optimization for Neural Architecture Search , 2022, 2204.03916.

[11]  F. Hutter,et al.  NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy , 2022, ICLR.

[12]  Mikhail Burtsev,et al.  A review of neural architecture search , 2021, Neurocomputing.

[13]  Luigi Nardi,et al.  LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso , 2021, AutoML.

[14]  M. Khodak,et al.  NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks , 2021, NeurIPS.

[15]  B. Bischl,et al.  YAHPO Gym - An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization , 2021, AutoML.

[16]  Bing Xue,et al.  BenchENAS: A Benchmarking Platform for Evolutionary Neural Architecture Search , 2021, IEEE Transactions on Evolutionary Computation.

[17]  P. Luo,et al.  Learning Versatile Neural Architectures by Propagating Network Codes , 2021, ICLR.

[18]  El-Ghazali Talbi,et al.  Automated Design of Deep Neural Networks , 2021, ACM Comput. Surv..

[19]  Mingkui Tan,et al.  Towards Accurate and Compact Architectures via Neural Architecture Transformer , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  X. Serra,et al.  FSD50K: An Open Dataset of Human-Labeled Sound Events , 2020, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[21]  B. Gabrys,et al.  NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Julien N. Siems,et al.  Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks , 2020, ICLR.

[23]  Longhui Wei,et al.  Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap , 2020, ACM Comput. Surv..

[24]  Krishna Teja Chitty-Venkata,et al.  Neural Architecture Search for Transformers: A Survey , 2022, IEEE Access.

[25]  G. Amer,et al.  Robust Speech Emotion Recognition Using CNN+LSTM Based on Stochastic Fractal Search Optimization Algorithm , 2022, IEEE Access.

[26]  J. Lerga,et al.  Detection of Non-Stationary GW Signals in High Noise From Cohen’s Class of Time–Frequency Representations Using Deep Learning , 2022, IEEE Access.

[27]  Babatounde Moctard Oloulade,et al.  Graph neural architecture search: A survey , 2022, Tsinghua Science and Technology.

[28]  B. Gabrys,et al.  Automated Deep Learning: Neural Architecture Search Is Not the End , 2021, ArXiv.

[29]  Emily Denton,et al.  Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research , 2021, NeurIPS Datasets and Benchmarks.

[30]  Frank Hutter,et al.  NAS-Bench-x11 and the Power of Learning Curves , 2021, NeurIPS.

[31]  Shinichi Shirakawa,et al.  NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters , 2021, ArXiv.

[32]  Aaron Klein,et al.  HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO , 2021, NeurIPS Datasets and Benchmarks.

[33]  Vinula Uthsara Buthgamumudalige,et al.  Neural Architecture Search for Generative Adversarial Networks: A Review , 2021, 2021 10th International Conference on Information and Automation for Sustainability (ICIAfS).

[34]  Hamza Ouarnoughi,et al.  Hardware-Aware Neural Architecture Search: Survey and Taxonomy , 2021, IJCAI.

[35]  Minghao Chen,et al.  AutoFormer: Searching Transformers for Visual Recognition , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[36]  A. Kai Qin,et al.  A Survey of Advances in Evolutionary Neural Architecture Search , 2021, 2021 IEEE Congress on Evolutionary Computation (CEC).

[37]  Javier Duarte,et al.  MLPerf Tiny Benchmark , 2021, NeurIPS Datasets and Benchmarks.

[38]  Xiaojun Chang,et al.  BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[39]  Marius Lindauer,et al.  Auto-Pytorch: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Fei Wang,et al.  Prioritized Architecture Sampling with Monto-Carlo Tree Search , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  T. Wirasingha,et al.  Automating Generative Adversarial Networks using Neural Architecture Search: A Review , 2021, 2021 International Conference on Emerging Smart Computing and Informatics (ESCI).

[42]  Wenwu Zhu,et al.  Automated Machine Learning on Graphs: A Survey , 2021, IJCAI.

[43]  Hamza Ouarnoughi,et al.  A Comprehensive Survey on Hardware-Aware Neural Architecture Search , 2021, ArXiv.

[44]  Nicholas D. Lane,et al.  Zero-Cost Proxies for Lightweight NAS , 2021, ICLR.

[45]  Tijmen Blankevoort,et al.  Distilling Optimal Neural Networks: Rapid Search in Diverse Spaces , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[46]  Nikola B. Kovachki,et al.  Fourier Neural Operator for Parametric Partial Differential Equations , 2020, ICLR.

[47]  Zhenguo Li,et al.  TransNAS-Bench-101: Improving transferability and Generalizability of Cross-Task Neural Architecture Search , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Joonsang Yu,et al.  DANCE: Differentiable Accelerator/Network Co-Exploration , 2020, 2021 58th ACM/IEEE Design Automation Conference (DAC).

[49]  Jianping Gou,et al.  Knowledge Distillation: A Survey , 2020, International Journal of Computer Vision.

[50]  Elliot J. Crowley,et al.  Neural Architecture Search without Training , 2020, ICML.

[51]  Yash Savani,et al.  Exploring the loss landscape in neural architecture search , 2020, UAI.

[52]  Colin White,et al.  BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search , 2019, AAAI.

[53]  Kaiyong Zhao,et al.  AutoML: A Survey of the State-of-the-Art , 2019, Knowl. Based Syst..

[54]  Samin Ishtiaq,et al.  NAS-Bench-ASR: Reproducible Neural Architecture Search for Speech Recognition , 2021, ICLR.

[55]  Detian Liu,et al.  Federated Neural Architecture Search Evolution and Open Problems: An Overview , 2021, BIC-TA.

[56]  Santanu Santra,et al.  Gradient Descent Effects on Differential Neural Architecture Search: A Survey , 2021, IEEE Access.

[57]  B. Gabrys,et al.  AutonoML: Towards an Integrated Framework for Autonomous Machine Learning , 2020, ArXiv.

[58]  Irwin King,et al.  AutoGraph: Automated Graph Neural Network , 2020, ICONIP.

[59]  Hangyu Zhu,et al.  From federated learning to federated neural architecture search: a survey , 2020, Complex & Intelligent Systems.

[60]  Nicholas D. Lane,et al.  BRP-NAS: Prediction-based NAS using GCNs , 2020, NeurIPS.

[61]  Jimeng Sun,et al.  HOLMES: Health OnLine Model Ensemble Serving for Deep Learning Models in Intensive Care Units , 2020, KDD.

[62]  Hong Yang,et al.  Graph Neural Architecture Search , 2020, IJCAI.

[63]  Evgeny Burnaev,et al.  NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing , 2020, IEEE Access.

[64]  Xiaojun Chang,et al.  A Comprehensive Survey of Neural Architecture Search , 2020, ACM Comput. Surv..

[65]  George Kyriakides,et al.  An Introduction to Neural Architecture Search for Convolutional Networks , 2020, ArXiv.

[66]  Tie-Yan Liu,et al.  Searching Better Architectures for Neural Machine Translation , 2020, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[67]  Badri Adhikari,et al.  A fully open-source framework for deep learning protein real-valued distances , 2020, Scientific Reports.

[68]  Yang Yu,et al.  A technical view on neural architecture search , 2020, International Journal of Machine Learning and Cybernetics.

[69]  Mengjie Zhang,et al.  Completely Automated CNN Architecture Design Based on Blocks , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[70]  Carole-Jean Wu,et al.  MLPerf: An Industry Standard Benchmark Suite for Machine Learning Performance , 2020, IEEE Micro.

[71]  Julien N. Siems,et al.  NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search , 2020, ICLR.

[72]  Yi Yang,et al.  NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search , 2020, ICLR.

[73]  Fabio Maria Carlucci,et al.  NAS evaluation is frustratingly hard , 2019, ICLR.

[74]  Cody Coleman,et al.  MLPerf Inference Benchmark , 2019, 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA).

[75]  Geoffrey I. Webb,et al.  ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels , 2019, Data Mining and Knowledge Discovery.

[76]  Cody A. Coleman,et al.  MLPerf Training Benchmark , 2019, MLSys.

[77]  Chuang Gan,et al.  Once for All: Train One Network and Specialize it for Efficient Deployment , 2019, ICLR.

[78]  Hao Chen,et al.  NAS-FCOS: Fast Neural Architecture Search for Object Detection , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[79]  Jiancheng Lv,et al.  Automatically Designing CNN Architectures Using Genetic Algorithm for Image Classification , 2018, ArXiv.

[80]  Mengjie Zhang,et al.  Evolving Deep Convolutional Neural Networks for Image Classification , 2017, IEEE Transactions on Evolutionary Computation.

[81]  S. Poornachandra.  Neural Architecture Search in Classical and Quantum Computers: A Survey , 2020 .

[82]  Marius Lindauer,et al.  Best Practices for Scientific Research on Neural Architecture Search , 2019, ArXiv.

[83]  Mohamed Saber Naceur,et al.  Reinforcement learning for neural architecture search: A review , 2019, Image Vis. Comput..

[84]  Forrest N. Iandola,et al.  SqueezeNAS: Fast Neural Architecture Search for Faster Semantic Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[85]  Keming Zhang,et al.  deepCR: Cosmic Ray Rejection with Deep Learning , 2019, J. Open Source Softw..

[86]  Jingtong Hu,et al.  When Neural Architecture Search Meets Hardware Implementation: from Hardware Awareness to Co-Design , 2019, 2019 IEEE Computer Society Annual Symposium on VLSI (ISVLSI).

[87]  Gaofeng Meng,et al.  RENAS: Reinforced Evolutionary Neural Architecture Search , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[88]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[89]  Aaron Klein,et al.  Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization , 2019, ArXiv.

[90]  Quoc V. Le,et al.  Searching for MobileNetV3 , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[91]  Martin Wistuba,et al.  A Survey on Neural Architecture Search , 2019, ArXiv.

[92]  Qi Tian,et al.  Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[93]  Aaron Klein,et al.  NAS-Bench-101: Towards Reproducible Neural Architecture Search , 2019, ICML.

[94]  Ameet Talwalkar,et al.  Random Search and Reproducibility for Neural Architecture Search , 2019, UAI.

[95]  Quoc V. Le,et al.  The Evolved Transformer , 2019, ICML.

[96]  Yuandong Tian,et al.  FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[97]  Kalyanmoy Deb,et al.  NSGA-Net: neural architecture search using multi-objective genetic algorithm , 2018, GECCO.

[98]  Song Han,et al.  ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware , 2018, ICLR.

[99]  Frank Hutter,et al.  Neural Architecture Search: A Survey , 2018, J. Mach. Learn. Res..

[100]  Yiming Yang,et al.  DARTS: Differentiable Architecture Search , 2018, ICLR.

[101]  Alok Aggarwal,et al.  Regularized Evolution for Image Classifier Architecture Search , 2018, AAAI.

[102]  Lars Kotthoff,et al.  Automated Machine Learning: Methods, Systems, Challenges , 2019, The Springer Series on Challenges in Machine Learning.

[103]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[104]  Isabelle Guyon,et al.  Taking Human out of Learning Applications: A Survey on Automated Machine Learning , 2018, 1810.13306.

[105]  Tie-Yan Liu,et al.  Neural Architecture Optimization , 2018, NeurIPS.

[106]  Leonidas J. Guibas,et al.  Taskonomy: Disentangling Task Transfer Learning , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[107]  Quoc V. Le,et al.  Efficient Neural Architecture Search via Parameter Sharing , 2018, ICML.

[108]  Max Welling,et al.  Spherical CNNs , 2018, ICLR.

[109]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[110]  Li Fei-Fei,et al.  Progressive Neural Architecture Search , 2017, ECCV.

[111]  Oriol Vinyals,et al.  Hierarchical Representations for Efficient Architecture Search , 2017, ICLR.

[112]  Frank Hutter,et al.  A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets , 2017, ArXiv.

[113]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[114]  Masanori Suganuma,et al.  A genetic programming approach to designing convolutional neural network architectures , 2017, GECCO.

[115]  Alan L. Yuille,et al.  Genetic CNN , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[116]  Quoc V. Le,et al.  Large-Scale Evolution of Image Classifiers , 2017, ICML.

[117]  Quoc V. Le,et al.  Neural Architecture Search with Reinforcement Learning , 2016, ICLR.

[118]  Richard Socher,et al.  Pointer Sentinel Mixture Models , 2016, ICLR.

[119]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[120]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[121]  Davide Anguita,et al.  Machine learning approaches for improving condition-based maintenance of naval propulsion plants , 2016 .

[122]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[123]  O. Troyanskaya,et al.  Predicting effects of noncoding variants with deep learning–based sequence model , 2015, Nature Methods.

[124]  Sanjeev Khudanpur,et al.  Librispeech: An ASR corpus based on public domain audio books , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[125]  Luís Torgo,et al.  OpenML: networked science in machine learning , 2014, SKDD.

[126]  Manfredo Atzori,et al.  Building the Ninapro database: A resource for the biorobotics community , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[127]  Carla Teixeira Lopes,et al.  TIMIT Acoustic-Phonetic Continuous Speech Corpus , 2012 .

[128]  Hans-Peter Kriegel,et al.  2D Image Registration in CT Images Using Radial Image Descriptors , 2011, MICCAI.

[129]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[130]  Max A. Little,et al.  Accurate telemonitoring of Parkinson’s disease progression by non-invasive speech tests , 2009 .

[131]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[132]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[133]  Ying Zhang,et al.  HMDB: the Human Metabolome Database , 2007, Nucleic Acids Res..

[134]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.