Erim Report Series Research in Management Auk: a Simple Alternative to the Auc Auk: a Simple Alternative to the Auc

AND KEYWORDS Abstract The area under Receiver Operating Characteristic (ROC) curve, also known as the AUC-index, is commonly used for ranking the performance of data mining models. The AUC has many merits, such as objectivity and ease of interpretation. However, since it is class indifferent, its usefulness while dealing with highly skewed data sets is questionable, to say the least. In this paper, we propose a simple alternative scalar measure to the AUCindex, the Area Under an Kappa curve (AUK). The proposed AUK-index compensates for the above basic flaw of the AUC by being sensitive to the class distribution. Therefore it is particularly suitable for measuring classifiers' performance on skewed data sets. After introducing the AUK we explore its mathematical relationship with the AUC and show that there is a nonlinear relation between them. Classifications The electronic versions of the papers in the ERIM report Series contain bibliographic metadata by the following classification systems: Abstract The area under Receiver Operating Characteristic (ROC) curve, also known as the AUC-index, is commonly used for ranking the performance of data mining models. The AUC has many merits, such as objectivity and ease of interpretation. However, since it is class indifferent, its usefulness while dealing with highly skewed data sets is questionable, to say the least. In this paper, we propose a simple alternative scalar measure to the AUC-index, the Area Under an Kappa curve (AUK). The proposed AUK-index compensates for the above basic flaw of the AUC by being sensitive to the class distribution. Therefore it is particularly suitable for measuring classifiers' performance on skewed data sets. After introducing the AUK we explore its mathematical relationship with the AUC and show that there is a nonlinear relation between them.

[1]  Gustavo E. A. P. A. Batista,et al.  A Survey on Graphical Methods for Classification Predictive Performance Evaluation , 2011, IEEE Transactions on Knowledge and Data Engineering.

[2]  J. Dul,et al.  Human factors: spanning the gap between OM and HRM , 2010 .

[3]  Wolfgang Ketter,et al.  Intelligent Personalized Trading Agents that Facilitate Real-Time Decisionmaking for Auctioneers and Buyers in the Dutch Flower Auctions , 2010 .

[4]  S. Knapp,et al.  Visualization of Ship Risk Profiles for the Shipping Industry , 2010 .

[5]  Niels A. H. Agatz,et al.  Sustainable Passenger Transportation: Dynamic Ride-Sharing , 2010 .

[6]  C. Chu,et al.  Linearization and Decomposition Methods for Large Scale Stochastic Inventory Routing Problem with Service Level Constraints , 2010 .

[7]  David J. Hand,et al.  Measuring classifier performance: a coherent alternative to the area under the ROC curve , 2009, Machine Learning.

[8]  Arie Ben-David,et al.  About the relationship between ROC curves and Cohen's kappa , 2008, Eng. Appl. Artif. Intell..

[9]  J. Demšar.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[10]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[11]  David J. Hand,et al.  A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems , 2001, Machine Learning.

[12]  Tom Fawcett,et al.  Robust Classification for Imprecise Environments , 2000, Machine Learning.

[13]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[14]  J. Fleiss,et al.  Statistical methods for rates and proportions , 1973 .

[15]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .