Multi-Objective Parameter Selection for Classifiers

Setting the free parameters of classifiers to different values can have a profound impact on their performance. For some methods, specialized tuning algorithms have been developed. These approaches mostly tune parameters according to a single criterion, such as the cross-validation error. However, it is sometimes desirable to obtain parameter values that optimize several concurrent - often conflicting - criteria. The TunePareto package provides a general and highly customizable framework to select optimal parameters for classifiers according to multiple objectives. Several strategies for sampling and optimizing parameters are supplied. The algorithm determines a set of Pareto-optimal parameter configurations and leaves the ultimate decision on the weighting of objectives to the researcher. Decision support is provided by novel visualization techniques.

[1]  A. E. Eiben,et al.  Introduction to Evolutionary Computing , 2003, Natural Computing Series.

[2]  Alex N. Kalos.  Automated neural network structure determination via discrete particle swarm optimization (for non-linear time series models) , 2005 .

[3]  Hans-Paul Schwefel,et al.  Evolution strategies – A comprehensive introduction , 2002, Natural Computing.

[4]  Harald Niederreiter,et al.  Random number generation and Quasi-Monte Carlo methods , 1992, CBMS-NSF regional conference series in applied mathematics.

[5]  R. Caflisch,et al.  Quasi-Monte Carlo integration , 1995 .

[6]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[7]  Christian Igel,et al.  Multi-Objective Optimization of Support Vector Machines , 2006, Multi-Objective Machine Learning.

[8]  Robert Sabourin,et al.  A PSO-based framework for dynamic SVM model selection , 2009, GECCO '09.

[9]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[10]  Max A. Little,et al.  Suitability of Dysphonia Measurements for Telemonitoring of Parkinson's Disease , 2008, IEEE Transactions on Biomedical Engineering.

[11]  Thomas Bartz-Beielstein,et al.  Sequential parameter optimization , 2005, 2005 IEEE Congress on Evolutionary Computation.

[12]  Massimo Pappalardo,et al.  Multiobjective Optimization: A Brief Overview , 2008 .

[13]  Ron Kohavi,et al.  Automatic Parameter Selection by Minimizing Estimated Error , 1995, ICML.

[14]  Richard Simon,et al.  Bias in error estimation when using cross-validation for model selection , 2006, BMC Bioinformatics.

[15]  Anne-Laure Boulesteix,et al.  Bmc Medical Research Methodology Open Access Optimal Classifier Selection and Negative Bias in Error Rate Estimation: an Empirical Study on High-dimensional Prediction , 2022 .

[16]  Andy Liaw,et al.  Classification and Regression by randomForest , 2007 .

[17]  Bernd Bischl,et al.  Resampling Methods in Model Validation , 2010 .

[18]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[19]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[20]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[21]  Kalyanmoy Deb,et al.  Multi-objective optimization using evolutionary algorithms , 2001, Wiley-Interscience series in systems and optimization.

[22]  H. Niederreiter.  Low-discrepancy and low-dispersion sequences , 1988 .

[23]  Hisao Ishibuchi,et al.  Evolutionary many-objective optimization: A short review , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[24]  Massimiliano Pontil,et al.  Properties of Support Vector Machines , 1998, Neural Computation.

[25]  Heike Trautmann,et al.  Integration of Preferences in Hypervolume-Based Multiobjective Evolutionary Algorithms by Means of Desirability Functions , 2010, IEEE Transactions on Evolutionary Computation.

[26]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[27]  Theodor J. Stewart,et al.  Multiple criteria decision analysis - an integrated approach , 2001 .

[28]  Guido Schwarzer,et al.  Easier parallel computing in R with snowfall and sfCluster , 2009, R J..

[29]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  Multiclass SVM Model Selection Using Particle Swarm Optimization , 2006, 2006 Sixth International Conference on Hybrid Intelligent Systems (HIS'06).

[30]  Thomas Bartz-Beielstein,et al.  Experimental Research in Evolutionary Computation - The New Experimentalism , 2010, Natural Computing Series.

[31]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[32]  Xiusheng Duan,et al.  Parameters optimization of Support Vector Machine based on Simulated Annealing and Genetic Algorithm , 2009, 2009 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[33]  A. Zell,et al.  Efficient parameter selection for support vector machines in classification and regression via model-based global optimization , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[34]  Christian Igel,et al.  Multi-objective Model Selection for Support Vector Machines , 2005, EMO.

[35]  Pedro M. Domingos,et al.  On the Optimality of the Simple Bayesian Classifier under Zero-One Loss , 1997, Machine Learning.

[36]  Gábor Csárdi,et al.  The igraph software package for complex network research , 2006 .

[37]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[38]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[39]  Jiao Licheng,et al.  Automatic parameters selection for SVM based on GA , 2004, Fifth World Congress on Intelligent Control and Automation (IEEE Cat. No.04EX788).

[40]  Yingqin Zhang.  Evolutionary Computation Based Automatic SVM Model Selection , 2008, 2008 Fourth International Conference on Natural Computation.

[41]  Paul Bratley,et al.  Algorithm 659: Implementing Sobol's quasirandom sequence generator , 1988, TOMS.