Evaluation of Classification Algorithms Using MCDM and Rank Correlation

Classification algorithm selection is an important issue in many disciplines. Since it normally involves more than one criterion, the task of algorithm selection can be modeled as multiple criteria decision making (MCDM) problems. Different MCDM methods evaluate classifiers from different aspects and thus they may produce divergent rankings of classifiers. The goal of this paper is to propose an approach to resolve disagreements among MCDM methods based on Spearman's rank correlation coefficient. Five MCDM methods are examined using 17 classification algorithms and 10 performance criteria over 11 public-domain binary classification datasets in the experimental study. The rankings of classifiers are quite different at first. After applying the proposed approach, the differences among MCDM rankings are largely reduced. The experimental results prove that the proposed approach can resolve conflicting MCDM rankings and reach an agreement among different MCDM methods.

[1]  Zhengxin Chen,et al.  A Multi-criteria Convex Quadratic Programming model for credit data analysis , 2008, Decis. Support Syst..

[2]  Gang Kou,et al.  Multiple factor hierarchical clustering algorithm for large scale web page and search engine clickstream data , 2012, Ann. Oper. Res..

[3]  Gang Kou,et al.  Analytic network process in risk assessment and decision analysis , 2014, Comput. Oper. Res..

[4]  A. S. Milani,et al.  Using different ELECTRE methods in strategic planning in the presence of human behavioral resistance , 2006, Adv. Decis. Sci..

[5]  Matthias Ehrgott,et al.  Multiple criteria decision analysis: state of the art surveys , 2005 .

[6]  Zhengxin Chen,et al.  A Descriptive Framework for the Field of Data Mining and Knowledge Discovery , 2008, Int. J. Inf. Technol. Decis. Mak..

[7]  อนิรุธ สืบสิงห์,et al.  Data Mining Practical Machine Learning Tools and Techniques , 2014 .

[8]  Yoon Ho Cho,et al.  Collaborative filtering with ordinal scale-based implicit ratings for mobile music recommendations , 2010, Inf. Sci..

[9]  Deng Ju-Long,et al.  Control problems of grey systems , 1982 .

[10]  Gwo-Hshiung Tzeng,et al.  Multicriteria Planning of Post‐Earthquake Sustainable Reconstruction , 2002 .

[11]  Honggang Wang,et al.  User preferences based software defect detection algorithms selection using MCDM , 2012, Inf. Sci..

[12]  Pat Langley,et al.  Induction of One-Level Decision Trees , 1992, ML.

[13]  Yi Peng,et al.  Ensemble of Software Defect Predictors: an AHP-Based Evaluation Method , 2011, Int. J. Inf. Technol. Decis. Mak..

[14]  Pedro M. Domingos,et al.  On the Optimality of the Simple Bayesian Classifier under Zero-One Loss , 1997, Machine Learning.

[15]  Taho Yang,et al.  The use of grey relational analysis in solving multiple attribute decision-making problems , 2008, Comput. Ind. Eng..

[16]  Aiko M. Hormann,et al.  Programs for Machine Learning. Part I , 1962, Inf. Control..

[17]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[18]  Ian H. Witten,et al.  Generating Accurate Rule Sets Without Global Optimization , 1998, ICML.

[19]  Gang Kou,et al.  A simple method to improve the consistency ratio of the pair-wise comparison matrix in ANP , 2011, Eur. J. Oper. Res..

[20]  Huan Neng Chiu,et al.  Vendor selection by integrated fuzzy MCDM techniques with independent and interdependent relationships , 2008, Inf. Sci..

[21]  Ludmila I. Kuncheva,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2004 .

[22]  Yi Peng,et al.  FAMCDM: A fusion approach of MCDM methods to rank multiclass classification algorithms , 2011 .

[23]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[24]  John R. Rice,et al.  The Algorithm Selection Problem , 1976, Adv. Comput..

[25]  José Hernández-Orallo,et al.  An experimental comparison of performance measures for classification , 2009, Pattern Recognit. Lett..

[26]  Lior Rokach,et al.  Ensemble-based classifiers , 2010, Artificial Intelligence Review.

[27]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[28]  Ching-Lai Hwang,et al.  Multiple Attribute Decision Making: Methods and Applications - A State-of-the-Art Survey , 1981, Lecture Notes in Economics and Mathematical Systems.

[29]  Yi Peng,et al.  Discovering Credit Cardholders’ Behavior by Multiple Criteria Linear Programming , 2005, Ann. Oper. Res..

[30]  Kate Smith-Miles,et al.  Cross-disciplinary perspectives on meta-learning for algorithm selection , 2009, CSUR.

[31]  Alexander Schnabl,et al.  Development of Multi-Criteria Metrics for Evaluation of Data Mining Algorithms , 1997, KDD.

[32]  Banu Soylu.  Integrating Prometheeii with the Tchebycheff Function for Multi Criteria Decision Making , 2010, Int. J. Inf. Technol. Decis. Mak..

[33]  David L. Olson,et al.  Comparison of weights in TOPSIS models , 2004, Math. Comput. Model..

[34]  Evangelos Triantaphyllou,et al.  The impact of aggregating benefit and cost criteria in four MCDA methods , 2005, IEEE Transactions on Engineering Management.

[35]  Bernard Roy,et al.  Aide multicritère à la décision : méthodes et cas , 1993 .

[36]  Geoffrey I. Webb.  Decision Tree Grafting From the All Tests But One Partition , 1999, IJCAI.

[37]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[38]  Ian Witten,et al.  Data Mining , 2000 .

[39]  Gwo-Hshiung Tzeng,et al.  A VIKOR-Based Multiple Criteria Decision Method for Improving Information Security Risk , 2009, Int. J. Inf. Technol. Decis. Mak..

[40]  Gwo-Hshiung Tzeng,et al.  Compromise solution by MCDM methods: A comparative analysis of VIKOR and TOPSIS , 2004, Eur. J. Oper. Res..

[41]  Bernard Roy,et al.  Classement et choix en présence de points de vue multiples , 1968 .

[42]  Robert C. Holte,et al.  Very Simple Classification Rules Perform Well on Most Commonly Used Datasets , 1993, Machine Learning.

[43]  Pankaj Gupta,et al.  Asset portfolio optimization using fuzzy mathematical programming , 2008, Inf. Sci..