Multi-objective evolutionary optimization for generating ensembles of classifiers in the ROC space

In this paper, we propose a novel approach for the multi-objective optimization of classifier ensembles in the ROC space. We first evolve a pool of simple classifiers with NSGA-II using values of the ROC curves as the optimization objectives. These simple classifiers are then combined at the decision level using the Iterative Boolean Combination method (IBC). This method produces multiple ensembles of classifiers optimized for various operating conditions. We perform a rigorous series of experiments to demonstrate the properties and behaviour of this approach. This allows us to propose interesting venues for future research on optimizing ensembles of classifiers using multi-objective evolutionary algorithms.

[1]  David Zhang,et al.  A New Framework for Adaptive Multimodal Biometrics Management , 2010, IEEE Transactions on Information Forensics and Security.

[2]  Mehryar Mohri,et al.  AUC Optimization vs. Error Rate Minimization , 2003, NIPS.

[3]  Ludmila I. Kuncheva,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2004 .

[4]  Marc Parizeau,et al.  DEAP: a python framework for evolutionary algorithms , 2012, GECCO '12.

[5]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[6]  Robert Sabourin,et al.  Overfitting cautious selection of classifier ensembles with genetic algorithms , 2009, Inf. Fusion.

[7]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[8]  Yves Lecourtier,et al.  A multi-model selection framework for unknown and/or evolutive misclassification cost problems , 2010, Pattern Recognit..

[9]  Peter A. Flach,et al.  Scored AUC Metrics for Classifier Evaluation and Selection , 2005 .

[10]  Dmitry O. Gorodnichy,et al.  Fusion of biometric systems using Boolean combination: an application to iris-based authentication , 2012, Int. J. Biom..

[11]  Michèle Sebag,et al.  ROC-Based Evolutionary Learning: Application to Medical Data Mining , 2003, Artificial Evolution.

[12]  Claudio Marrocco,et al.  On Linear Combinations of Dichotomizers for Maximizing the Area Under the ROC Curve , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[13]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Robert Sabourin,et al.  Adaptive ROC-based ensembles of HMMs applied to anomaly detection , 2012, Pattern Recognit..

[15]  Mark Johnston,et al.  AUC analysis of the pareto-front using multi-objective GP for classification with unbalanced data , 2010, GECCO '10.

[16]  Mark Johnston,et al.  Evolving ensembles in multi-objective genetic programming for classification with unbalanced data , 2011, GECCO '11.

[17]  Peter A. Flach,et al.  Learning Decision Trees Using the Area Under the ROC Curve , 2002, ICML.

[18]  Henrik Boström,et al.  Maximizing the Area under the ROC Curve using Incremental Reduced Error Pruning , 2005, ICML 2005.

[19]  Joo-Hwee Lim,et al.  An ensemble classifier learning approach to ROC optimization , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[20]  Raymond N. J. Veldhuis,et al.  Threshold-optimized decision-level fusion and its application to biometrics , 2009, Pattern Recognit..

[21]  Marco Laumanns,et al.  SPEA2: Improving the strength pareto evolutionary algorithm , 2001 .

[22]  Shaomin Wu,et al.  A scored AUC Metric for Classifier Evaluation and Selection , 2005 .

[23]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[24]  Jonathan E. Fieldsend,et al.  Multi-class ROC analysis from a multi-objective optimisation perspective , 2006, Pattern Recognit. Lett..

[25]  Tom Fawcett,et al.  Robust Classification for Imprecise Environments , 2000, Machine Learning.

[26]  Robert Sabourin,et al.  Iterative Boolean combination of classifiers in the ROC space: An application to anomaly detection with HMMs , 2010, Pattern Recognit..