Speeding up algorithm selection using average ranking and active testing by introducing runtime

Algorithm selection methods can be speeded-up substantially by incorporating multi-objective measures that give preference to algorithms that are both promising and fast to evaluate. In this paper, we introduce such a measure, A3R, and incorporate it into two algorithm selection techniques: average ranking and active testing. Average ranking combines algorithm rankings observed on prior datasets to identify the best algorithms for a new dataset. The aim of the second method is to iteratively select algorithms to be tested on the new dataset, learning from each new evaluation to intelligently select the next best candidate. We show how both methods can be upgraded to incorporate a multi-objective measure A3R that combines accuracy and runtime. It is necessary to establish the correct balance between accuracy and runtime, as otherwise time will be wasted by conducting less informative tests. The correct balance can be set by an appropriate parameter setting within function A3R that trades off accuracy and runtime. Our results demonstrate that the upgraded versions of Average Ranking and Active Testing lead to much better mean interval loss values than their accuracy-based counterparts.

[1]  F. Hutter,et al.  Hydra-MIP : Automated Algorithm Configuration and Selection for Mixed Integer Programming , 2011 .

[2]  Frank Hutter,et al.  Initializing Bayesian Hyperparameter Optimization via Meta-Learning , 2015, AAAI.

[3]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[4]  Ricardo Vilalta,et al.  Metalearning - Applications to Data Mining , 2008, Cognitive Technologies.

[5]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[6]  Luís Torgo,et al.  OpenML: networked science in machine learning , 2014, SKDD.

[7]  Pavel Brazdil,et al.  Measures for Combining Accuracy and Time for Meta-learning , 2014, MetaSel@ECAI.

[8]  Quan Sun,et al.  Pairwise meta-rules for better meta-learning-based algorithm ranking , 2013, Machine Learning.

[9]  John R. Rice,et al.  The Algorithm Selection Problem , 1976, Adv. Comput..

[10]  Holger H. Hoos,et al.  Automated Algorithm Configuration and Parameter Tuning , 2012, Autonomous Search.

[11]  Hilan Bensusan,et al.  Tell me who can learn you and I can tell you who you are: Landmarking Various Learning Algorithms , 2000 .

[12]  Carlos Soares,et al.  A Comparison of Ranking Methods for Classification Algorithm Selection , 2000, ECML.

[13]  Joaquin Vanschoren,et al.  Selecting Classification Algorithms with Active Testing , 2012, MLDM.

[14]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[15]  Carlos Soares,et al.  Ranking Learning Algorithms: Using IBL and Meta-Learning on Accuracy and Time Results , 2003, Machine Learning.

[16]  Wei Chu,et al.  A contextual-bandit approach to personalized news article recommendation , 2010, WWW '10.

[17]  Kate Smith-Miles,et al.  Cross-disciplinary perspectives on meta-learning for algorithm selection , 2009, CSUR.

[18]  Vasyl Pihur,et al.  RankAggreg, an R package for weighted rank aggregation , 2009, BMC Bioinformatics.

[19]  Teresa Bernarda Ludermir,et al.  Active Selection of Training Examples for Meta-Learning , 2007, 7th International Conference on Hybrid Intelligent Systems (HIS 2007).

[20]  K. McConway.  Distribution-free Tests, H.R. Neave, P.L. Worthington. Unwin Hyman, London (1988), xvi, +430. Price £40.00 hardback, £14.95 paperback , 1989 .

[21]  Johannes Fürnkranz,et al.  An Evaluation of Landmarking Variants , 2001 .

[22]  Ya Zhang,et al.  Active Learning for Ranking through Expected Loss Optimization , 2010, IEEE Transactions on Knowledge and Data Engineering.

[23]  Norbert Jankowski,et al.  Complexity Measures for Meta-learning and Their Optimality , 2011, Algorithmic Probability and Friends.

[24]  W. J. Studden,et al.  Theory Of Optimal Experiments , 1972 .

[25]  Pavel Brazdil,et al.  Active Testing Strategy to Predict the Best Classification Algorithm via Sampling and Metalearning , 2010, ECAI.

[26]  Iain D. Currie,et al.  Distribution‐Free Tests , 1989 .

[27]  Joaquin Vanschoren,et al.  Fast Algorithm Selection Using Learning Curves , 2015, IDA.

[28]  Shili Lin,et al.  Rank aggregation methods , 2010 .

[29]  Joaquin Vanschoren,et al.  Selecting Classification Algorithms with Active Testing on Similar Datasets , 2012 .