A Literature Survey and Empirical Study of Meta-Learning for Classifier Selection

Classification is the key and most widely studied paradigm in machine learning community. The selection of appropriate classification algorithm for a particular problem is a challenging task, formally known as algorithm selection problem (ASP) in literature. It is increasingly becoming focus of research in machine learning community. Meta-learning has demonstrated substantial success in solving ASP, especially in the domain of classification. Considerable progress has been made in classification algorithm recommendation and researchers have proposed various methods in literature that tackles ASP in many different ways in meta-learning setup. Yet there is a lack of survey and comparative study that critically analyze, summarize and assess the performance of existing methods. To fill these gaps, in this paper we first present a literature survey of classification algorithm recommendation methods. The survey shed light on the motivational reasons for pursuing classifier selection through meta-learning and comprehensively discusses the different phases of classifier selection based on a generic framework that is formed as an outcome of reviewing prior works. Subsequently, we critically analyzed and summarized the existing studies from the literature in three important dimensions i.e., meta-features, meta-learner and meta-target. In the second part of this paper, we present extensive comparative evaluation of all the prominent methods for classifier selection based on 17 classification algorithms and 84 benchmark datasets. The comparative study quantitatively assesses the performance of classifier selection methods and highlight the limitations and strengths of meta-features, meta-learners and meta-target in classification algorithm recommendation system. Finally, we conclude this paper by identifying current challenges and suggesting future work directions. We expect that this work will provide baseline and a solid overview of state of the art works in this domain to new researchers, and will steer future research in this direction.

[1]  Marius Thomas Lindauer,et al.  AutoFolio: An Automatically Configured Algorithm Selector , 2015, J. Artif. Intell. Res..

[2]  Hilan Bensusan,et al.  Estimating the Predictive Accuracy of a Classifier , 2001, ECML.

[3]  Hilan Bensusan,et al.  Meta-Learning by Landmarking Various Learning Algorithms , 2000, ICML.

[4]  Hendrik Blockeel,et al.  A new way to share, organize and learn from experiments , 2012 .

[5]  Heike Trautmann,et al.  Automated Algorithm Selection: Survey and Perspectives , 2018, Evolutionary Computation.

[6]  Eunho Yang,et al.  Learning to Propagate Labels: Transductive Propagation Network for Few-Shot Learning , 2018, ICLR.

[7]  Ana Carolina Lorena,et al.  Data complexity meta-features for regression problems , 2017, Machine Learning.

[8]  Bernd Bischl,et al.  To tune or not to tune: Recommending when to adjust SVM hyper-parameters via meta-learning , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[9]  João Mendes-Moreira,et al.  Towards Automatic Generation of Metafeatures , 2016, PAKDD.

[10]  Huei Diana Lee,et al.  Metalearning for choosing feature selection algorithms in data mining: Proposal of a new framework , 2017, Expert Syst. Appl..

[11]  Adriano Rivolli,et al.  Characterizing classification datasets: a study of meta-features for meta-learning. , 2018 .

[12]  Jong-Seok Lee,et al.  A meta-learning approach for determining the number of clusters with consideration of nearest neighbors , 2013, Inf. Sci..

[13]  Elisa Guerrero Vázquez,et al.  Multiple comparison procedures applied to model selection , 2002, Neurocomputing.

[14]  Xiaoyan Zhu,et al.  A new classification algorithm recommendation method based on link prediction , 2018, Knowl. Based Syst..

[15]  Verónica Bolón-Canedo,et al.  Can classification performance be predicted by complexity measures? A study using microarray data , 2017, Knowledge and Information Systems.

[16]  Aaron Klein,et al.  Efficient and Robust Automated Machine Learning , 2015, NIPS.

[17]  Frank Hutter,et al.  Initializing Bayesian Hyperparameter Optimization via Meta-Learning , 2015, AAAI.

[18]  Tin Kam Ho,et al.  Complexity Measures of Supervised Classification Problems , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  MetaStream: A meta-learning based method for periodic algorithm selection in time-changing data , 2014, Neurocomputing.

[20]  Martial Hebert,et al.  Low-Shot Learning from Imaginary Data , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[22]  Sergey Levine,et al.  Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks , 2017, ICML.

[23]  João Gama,et al.  Characterizing the Applicability of Classification Algorithms Using Meta-Level Learning , 1994, ECML.

[24]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  Noise detection in the meta-learning level , 2016, Neurocomputing.

[25]  George D. C. Cavalcanti,et al.  Online local pool generation for dynamic classifier selection: an extended version , 2018, Pattern Recognit..

[26]  Bogdan Gabrys,et al.  Meta-learning for time series forecasting and forecast combination , 2010, Neurocomputing.

[27]  Qinbao Song,et al.  A Generic Multilabel Learning-Based Classification Algorithm Recommendation Method , 2014, TKDD.

[28]  Yi Yang,et al.  Compound Memory Networks for Few-Shot Video Classification , 2018, ECCV.

[29]  Teresa Bernarda Ludermir,et al.  Active learning and data manipulation techniques for generating training examples in meta-learning , 2016, Neurocomputing.

[30]  Michèle Sebag,et al.  Alors: An algorithm recommender system , 2017, Artif. Intell..

[31]  Francisco Herrera,et al.  Predicting noise filtering efficacy with data complexity measures for nearest neighbor classification , 2013, Pattern Recognit..

[32]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  Meta-learning to select the best meta-heuristic for the Traveling Salesman Problem: A comparison of meta-features , 2016, Neurocomputing.

[33]  Hans Degroote.  Online Algorithm Selection , 2017, IJCAI.

[34]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[35]  Myra Spiliopoulou,et al.  NOEMON: An Intelligent Assistant for Classifier Selection , 2007 .

[36]  Teresa Bernarda Ludermir,et al.  Meta-learning approaches to selecting time series models , 2004, Neurocomputing.

[37]  Kate Smith-Miles,et al.  Instance spaces for machine learning classification , 2017, Machine Learning.

[38]  Sungyoung Lee,et al.  Accurate multi-criteria decision making methodology for recommending machine learning algorithm , 2017, Expert Syst. Appl..

[39]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  Effects of Random Sampling on SVM Hyper-parameter Tuning , 2016, ISDA.

[40]  Christophe G. Giraud-Carrier,et al.  Automatic selection of classification learning algorithms for data mining practitioners , 2013, Intell. Data Anal..

[41]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  A new data characterization for selecting clustering algorithms using meta-learning , 2019, Inf. Sci..

[42]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[43]  Bogdan Gabrys,et al.  Metalearning: a survey of trends and technologies , 2013, Artificial Intelligence Review.

[44]  Qinbao Song,et al.  Automatic recommendation of classification algorithms based on data set characteristics , 2012, Pattern Recognit..

[45]  Can Cui,et al.  A recommendation system for meta-modeling: A meta-learning based approach , 2016, Expert Syst. Appl..

[46]  Joaquin Vanschoren,et al.  Speeding up algorithm selection using average ranking and active testing by introducing runtime , 2018, Machine Learning.

[47]  Christophe G. Giraud-Carrier,et al.  Predicting Algorithm Accuracy with a Small Set of Effective Meta-Features , 2008, 2008 Seventh International Conference on Machine Learning and Applications.

[48]  Antonio González Muñoz,et al.  A Set of Complexity Measures Designed for Applying Meta-Learning to Instance Selection , 2015, IEEE Transactions on Knowledge and Data Engineering.

[49]  George D. C. Cavalcanti,et al.  META-DES.Oracle: Meta-learning and feature selection for dynamic ensemble selection , 2017, Inf. Fusion.

[50]  Christophe G. Giraud-Carrier,et al.  The data mining advisor: meta-learning at the service of practitioners , 2005, Fourth International Conference on Machine Learning and Applications (ICMLA'05).

[51]  Hilan Bensusan,et al.  Discovering Task Neighbourhoods Through Landmark Learning Performances , 2000, PKDD.

[52]  Andreas Dengel,et al.  Automatic classifier selection for non-experts , 2012, Pattern Analysis and Applications.

[53]  Qinbao Song,et al.  An improved data characterization method and its application in classification algorithm recommendation , 2015, Applied Intelligence.

[54]  Francisco Herrera,et al.  An automatic extraction method of the domains of competence for learning classifiers using data complexity measures , 2013, Knowledge and Information Systems.

[55]  Lars Kotthoff,et al.  Algorithm Selection for Combinatorial Search Problems: A Survey , 2012, AI Mag..

[56]  George D. C. Cavalcanti,et al.  META-DES: A dynamic ensemble selection framework using meta-learning , 2015, Pattern Recognit..

[57]  Leandro Nunes de Castro,et al.  Clustering algorithm selection by meta-learning systems: A new distance-based problem characterization and ranking combination methods , 2015, Inf. Sci..

[58]  Kate Smith-Miles,et al.  Cross-disciplinary perspectives on meta-learning for algorithm selection , 2009, CSUR.

[59]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[60]  Nikolaj Tatti,et al.  Distances between Data Sets Based on Summary Statistics , 2007, J. Mach. Learn. Res..

[61]  Carlos Soares,et al.  Ranking Learning Algorithms: Using IBL and Meta-Learning on Accuracy and Time Results , 2003, Machine Learning.

[62]  John R. Rice,et al.  The Algorithm Selection Problem , 1976, Adv. Comput..

[63]  Stan Matwin,et al.  Ensembles of label noise filters: a ranking approach , 2016, Data Mining and Knowledge Discovery.

[64]  Jens Lehmann,et al.  How Complex Is Your Classification Problem? , 2018, ACM Comput. Surv..

[65]  Bo Zhang,et al.  An instance-based learning recommendation algorithm of imbalance handling methods , 2019, Appl. Math. Comput..

[66]  Lars Kotthoff,et al.  Automated Machine Learning: Methods, Systems, Challenges , 2019, The Springer Series on Challenges in Machine Learning.

[67]  Geoff Holmes,et al.  The online performance estimation framework: heterogeneous ensemble learning for data streams , 2017, Machine Learning.

[68]  Bernd Bischl,et al.  ASlib: A benchmark library for algorithm selection , 2015, Artif. Intell..

[69]  Yuming Zhou,et al.  A Feature Subset Selection Algorithm Automatic Recommendation Method , 2013, J. Artif. Intell. Res..

[70]  Ricardo Vilalta,et al.  A Perspective View and Survey of Meta-Learning , 2002, Artificial Intelligence Review.

[71]  Hilan Bensusan,et al.  God Doesn't Always Shave with Occam's Razor - Learning When and How to Prune , 1998, ECML.

[72]  Victor Ciesielski,et al.  Matching Data Mining Algorithm Suitability to Data Characteristics Using a Self-Organizing Map , 2001, HIS.

[73]  Abraham Bernstein,et al.  A survey of intelligent assistants for data analysis , 2013, CSUR.

[74]  Linyuan Lu,et al.  Link prediction based on local random walk , 2010, 1001.2467.

[75]  Mario A. Muñoz,et al.  Algorithm selection for black-box continuous optimization problems: A survey on methods and challenges , 2015, Inf. Sci..

[76]  Zhi-Hua Zhou,et al.  ML-KNN: A lazy learning approach to multi-label learning , 2007, Pattern Recognit..

[77]  Peter A. Flach,et al.  Decision Tree-Based Data Characterization for Meta-Learning , 2002 .

[78]  George D. C. Cavalcanti,et al.  Dynamic classifier selection: Recent advances and perspectives , 2018, Inf. Fusion.