Amazon SageMaker Autopilot: a white box AutoML solution at scale

We present Amazon SageMaker Autopilot: a fully managed system that provides an automatic machine learning solution. Given a tabular dataset and the target column name, Autopilot identifies the problem type, analyzes the data and produces a diverse set of complete ML pipelines, which are tuned to generate a leaderboard of candidate models that the customer can choose from. The diversity allows users to balance between different needs such as model accuracy vs. latency. By exposing not only the final models but the way they are trained, meaning the pipelines, we allow to customize the generated training pipeline, thus catering the need of users with different levels of expertise. This trait is crucial for users and is the main novelty of Autopilot; it provides a solution that on one hand is not fully black-box and can be further worked on, while on the other hand is not a do it yourself solution, requiring expertise in all aspects of machine learning. This paper describes the different components in the eco-system of Autopilot, emphasizing the infrastructure choices that allow scalability, high quality models, editable ML pipelines, consumption of artifacts of offline meta-learning, and a convenient integration with the entire SageMaker system allowing these trained models to be used in a production setting.

[1]  Krishnaram Kenthapadi,et al.  Fair Bayesian Optimization , 2020, ArXiv.

[2]  David Salinas,et al.  A Quantile-based Approach for Hyperparameter Transfer Learning , 2020, ICML.

[3]  Aaron Klein,et al.  Efficient and Robust Automated Machine Learning , 2015, NIPS.

[4]  John Myles White,et al.  Bandit Algorithms for Website Optimization , 2012 .

[5]  Bernd Bischl,et al.  Meta learning for defaults: symbolic defaults , 2018, ICONIP 2018.

[6]  Chris Eliasmith,et al.  Hyperopt: a Python library for model selection and hyperparameter optimization , 2015 .

[7]  Rich Caruana,et al.  Ensemble selection from libraries of models , 2004, ICML.

[8]  Hang Zhang,et al.  AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data , 2020, ArXiv.

[9]  Kaiyong Zhao,et al.  AutoML: A Survey of the State-of-the-Art , 2019, Knowl. Based Syst..

[10]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[11]  Alessandro Lazaric,et al.  A single algorithm for both restless and rested rotting bandits , 2020, AISTATS.

[12]  Joaquin Vanschoren,et al.  GAMA: Genetic Automated Machine learning Assistant , 2019, J. Open Source Softw..

[13]  Lars Schmidt-Thieme,et al.  Sequential Model-Free Hyperparameter Tuning , 2015, 2015 IEEE International Conference on Data Mining.

[14]  MullerMichael,et al.  Human-AI Collaboration in Data Science , 2019 .

[15]  Cedric Archambeau,et al.  Cost-aware Bayesian Optimization , 2020, ArXiv.

[16]  Aaron Klein,et al.  BOHB: Robust and Efficient Hyperparameter Optimization at Scale , 2018, ICML.

[17]  Valerio Perrone,et al.  Pareto-efficient Acquisition Functions for Cost-Aware Bayesian Optimization , 2020, ArXiv.

[18]  Peter Auer,et al.  The Nonstochastic Multiarmed Bandit Problem , 2002, SIAM J. Comput..

[19]  Eduardo F. Morales,et al.  An Introduction to Reinforcement Learning , 2011 .

[20]  Marco F. Huber,et al.  Benchmark and Survey of Automated Machine Learning Frameworks. , 2019 .

[21]  Luís Torgo,et al.  OpenML: networked science in machine learning , 2014, SKDD.

[22]  Aaron Klein,et al.  Auto-sklearn: Efficient and Robust Automated Machine Learning , 2019, Automated Machine Learning.

[23]  Gilles Louppe,et al.  Independent consultant , 2013 .

[24]  Qingquan Song,et al.  Auto-Keras: An Efficient Neural Architecture Search System , 2018, KDD.

[25]  Matthias W. Seeger,et al.  Scalable Hyperparameter Transfer Learning , 2018, NeurIPS.

[26]  Cedric Archambeau,et al.  Constrained Bayesian Optimization with Max-Value Entropy Search , 2019, ArXiv.

[27]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[28]  C. Archambeau,et al.  Multiple Adaptive Bayesian Linear Regression for Scalable Bayesian Optimization with Warm Start , 2017, 1712.02902.

[29]  Lars Schmidt-Thieme,et al.  Learning hyperparameter optimization initializations , 2015, 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[30]  Matthias Seeger,et al.  Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning , 2019, NeurIPS.

[31]  Marco F. Huber,et al.  Survey on Automated Machine Learning , 2019, ArXiv.

[32]  Fela Winkelmolen,et al.  Practical and sample efficient zero-shot HPO , 2020, ArXiv.

[33]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[34]  A. Azzouz.  2011 , 2020, City.

[35]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[36]  Joaquin Vanschoren,et al.  Meta-Learning: A Survey , 2018, Automated Machine Learning.

[37]  Randal S. Olson,et al.  TPOT: A Tree-based Pipeline Optimization Tool for Automating Machine Learning , 2016, AutoML@ICML.