Multi-class ROC analysis from a multi-objective optimisation perspective

The receiver operating characteristic (ROC) has become a standard tool for the analysis and comparison of classifiers when the costs of misclassification are unknown. There has been relatively little work, however, examining ROC for more than two classes. Here we discuss and present an extension to the standard two-class ROC for multi-class problems. We define the ROC surface for the Q-class problem in terms of a multi-objective optimisation problem in which the goal is to simultaneously minimise the Q(Q-1) misclassification rates, when the misclassification costs and parameters governing the classifier's behaviour are unknown. We present an evolutionary algorithm to locate the Pareto front-the optimal trade-off surface between misclassifications of different types. The use of the Pareto optimal surface to compare classifiers is discussed and we present a straightforward multi-class analogue of the Gini coefficient. The performance of the evolutionary algorithm is illustrated on a synthetic three class problem, for both k-nearest neighbour and multi-layer perceptron classifiers.

[1]  Peter A. Flach,et al.  Decision Support for Data Mining , 2003 .

[2]  Matthew A. Kupinski,et al.  Multiobjective Genetic Optimization of Diagnostic Classifiers with Implications for Generating ROC Curves , 1999, IEEE Trans. Medical Imaging.

[3]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[4]  Marc Alexa,et al.  Computing and Rendering Point Set Surfaces , 2003, IEEE Trans. Vis. Comput. Graph..

[5]  Michael E. Tipping,et al.  Shadow targets: A novel algorithm for topographic projections by radial basis functions , 1998, Neurocomputing.

[6]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[7]  Niall M. Adams,et al.  Comparing classifiers when the misallocation costs are uncertain , 1999, Pattern Recognit..

[8]  Tom Fawcett,et al.  Analysis and Visualization of Classifier Performance: Comparison under Imprecise Class and Cost Distributions , 1997, KDD.

[9]  Robert M. Nishikawa,et al.  Optimization and FROC analysis of rule-based detection schemes using a multiobjective approach , 1998, IEEE Transactions on Medical Imaging.

[10]  Jonathan E. Fieldsend,et al.  Visualisation of multi-class ROC surfaces , 2005 .

[11]  Mahesan Niranjan,et al.  Parcel: Feature Subset Selection in Variable Cost Domains , 1998 .

[12]  M. Zweig,et al.  Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. , 1993, Clinical chemistry.

[13]  Peter A. Flach,et al.  ROC Analysis in Artificial Intelligence, 1st International Workshop, ROCAI-2004, Valencia, Spain, August 22, 2004 , 2004, ROCAI.

[14]  Jonathan E. Fieldsend,et al.  Multiobjective optimization of safety related systems: an application to short-term conflict alert , 2006, IEEE Transactions on Evolutionary Computation.

[15]  C. A. Coello Coello,et al.  A Comprehensive Survey of Evolutionary-Based Multiobjective Optimization Techniques , 1999, Knowledge and Information Systems.

[16]  D. Mossman.  Three-way ROCs , 1999, Medical decision making : an international journal of the Society for Medical Decision Making.

[17]  David J. Hand,et al.  A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems , 2001, Machine Learning.

[18]  C. Holmes,et al.  A probabilistic nearest neighbour method for statistical pattern recognition , 2002 .

[19]  Niall M. Adams,et al.  Likelihood inference in nearest‐neighbour classification models , 2003 .

[20]  Jianqing Fan,et al.  Local polynomial modelling and its applications , 1994 .

[21]  David W. Corne,et al.  Approximating the Nondominated Front Using the Pareto Archived Evolution Strategy , 2000, Evolutionary Computation.

[22]  Gary B. Lamont,et al.  Multiobjective Evolutionary Algorithms: Analyzing the State-of-the-Art , 2000, Evolutionary Computation.

[23]  Dunja Mladenic,et al.  Data mining and decision support : integration and collaboration , 2003 .

[24]  Trevor Hastie,et al.  Neural Networks and Related Methods for Classification - Discussion , 1994 .

[25]  Michael E. Tipping,et al.  Feed-forward neural networks and topographic mappings for exploratory data analysis , 1996, Neural Computing & Applications.

[26]  M. Anastasio,et al.  Multiobjective genetic optimization of diagnostic classifiers with implications for generating receiver operating characteristic curves , 1999, IEEE Transactions on Medical Imaging.

[27]  Jonathan E. Fieldsend,et al.  Using unconstrained elite archives for multiobjective optimization , 2003, IEEE Trans. Evol. Comput..

[28]  Xin Yao,et al.  Evolutionary programming made faster , 1999, IEEE Trans. Evol. Comput..

[29]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[30]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[31]  Konrad Polthier,et al.  Anisotropic fairing of point sets , 2005 .

[32]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[33]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[34]  Jonathan E. Fieldsend,et al.  Formulation and comparison of multi-class ROC surfaces , 2005 .

[35]  Jonathan E. Fieldsend,et al.  ROC Optimisation of Safety Related Systems , 2004, ROCAI.

[36]  Tom Fawcett,et al.  Robust Classification Systems for Imprecise Environments , 1998, AAAI/IAAI.