DarwinML: A Graph-based Evolutionary Algorithm for Automated Machine Learning

As an emerging field, Automated Machine Learning (AutoML) aims to reduce or eliminate manual operations that require expertise in machine learning. In this paper, a graph-based architecture is employed to represent flexible combinations of ML models, which provides a large searching space compared to tree-based and stacking-based architectures. Based on this, an evolutionary algorithm is proposed to search for the best architecture, where the mutation and heredity operators are the key for architecture evolution. With Bayesian hyper-parameter optimization, the proposed approach can automate the workflow of machine learning. On the PMLB dataset, the proposed approach shows the state-of-the-art performance compared with TPOT, Autostacker, and auto-sklearn. Some of the optimized models are with complex structures which are difficult to obtain in manual design.

[1]  Sergio Escalera,et al.  A brief Review of the ChaLearn AutoML Challenge: Any-time Any-dataset Learning without Human Intervention , 2016, AutoML@ICML.

[2]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[3]  Gisele L. Pappa,et al.  RECIPE: A Grammar-Based Framework for Automatically Evolving Classification Pipelines , 2017, EuroGP.

[4]  Randal S. Olson,et al.  Automating Biomedical Data Science Through Tree-Based Pipeline Optimization , 2016, EvoApplications.

[5]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[6]  Scott M. Williams,et al.  A balanced accuracy function for epistasis modeling in imbalanced datasets using multifactor dimensionality reduction , 2007, Genetic epidemiology.

[7]  Aaron Klein,et al.  Efficient and Robust Automated Machine Learning , 2015, NIPS.

[8]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[9]  Krzysztof Krawiec,et al.  Semantic Backpropagation for Designing Search Operators in Genetic Programming , 2015, IEEE Transactions on Evolutionary Computation.

[10]  Randal S. Olson,et al.  PMLB: a large benchmark suite for machine learning evaluation and comparison , 2017, BioData Mining.

[11]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[12]  Quoc V. Le,et al.  Large-Scale Evolution of Image Classifiers , 2017, ICML.

[13]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[14]  Randal S. Olson,et al.  Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science , 2016, GECCO.

[15]  Julian Francis Miller,et al.  Cartesian genetic programming , 2010, GECCO.

[16]  Hugo Jair Escalante,et al.  EvoDAG: A semantic Genetic Programming Python library , 2016, 2016 IEEE International Autumn Meeting on Power, Electronics and Computing (ROPEC).

[17]  Nando de Freitas,et al.  A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning , 2010, ArXiv.

[18]  Conor Ryan,et al.  Handbook of Genetic Programming Applications , 2015, Springer International Publishing.

[19]  John R. Koza,et al.  Genetic programming 2 - automatic discovery of reusable programs , 1994, Complex Adaptive Systems.

[20]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[21]  Luís Torgo,et al.  OpenML: networked science in machine learning , 2014, SKDD.

[22]  Hod Lipson,et al.  Autostacker: a compositional evolutionary learning system , 2018, GECCO.

[23]  A. Asuncion,et al.  UCI Machine Learning Repository, University of California, Irvine, School of Information and Computer Sciences , 2007 .

[24]  Hugo Jair Escalante,et al.  Semantic Genetic Programming for Sentiment Analysis , 2015, NEO.

[25]  Daniel A. Ashlock,et al.  Evolving fractal art with a directed acyclic graph genetic programming representation , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[26]  Kalyanmoy Deb,et al.  A Comparative Analysis of Selection Schemes Used in Genetic Algorithms , 1990, FOGA.

[27]  Tobias Glasmachers,et al.  Limits of End-to-End Learning , 2017, ACML.

[28]  Nichael Lynn Cramer,et al.  A Representation for the Adaptive Generation of Simple Sequential Programs , 1985, ICGA.

[29]  A. B. Kahn,et al.  Topological sorting of large networks , 1962, CACM.

[30]  Udayan Khurana,et al.  Automating Feature Engineering , 2016 .

[31]  Christian Igel,et al.  Evolutionary tuning of multiple SVM parameters , 2005, ESANN.

[32]  Li Fei-Fei,et al.  Progressive Neural Architecture Search , 2017, ECCV.

[33]  Artus Krohn-Grimberghe,et al.  AutoCompete: A Framework for Machine Learning Competition , 2015, ArXiv.

[34]  Lars Kotthoff,et al.  Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA , 2017, J. Mach. Learn. Res..

[35]  Sergio Escalera,et al.  Design of the 2015 ChaLearn AutoML challenge , 2015, IJCNN.

[36]  Deepak S. Turaga,et al.  Feature Engineering for Predictive Modeling using Reinforcement Learning , 2017, AAAI.

[37]  Pavel Kordík,et al.  Discovering predictive ensembles for transfer learning and meta-learning , 2017, Machine Learning.

[38]  A. A. Ghorbani,et al.  Stacked generalization in neural networks: generalization on statistically neutral problems , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).