A review on the combination of binary classifiers in multiclass problems

Several real problems involve the classification of data into categories or classes. Given a data set containing data whose classes are known, Machine Learning algorithms can be employed for the induction of a classifier able to predict the class of new data from the same domain, performing the desired discrimination. Some learning techniques are originally conceived for the solution of problems with only two classes, also named binary classification problems. However, many problems require the discrimination of examples into more than two categories or classes. This paper presents a survey on the main strategies for the generalization of binary classifiers to problems with more than two classes, known as multiclass classification problems. The focus is on strategies that decompose the original multiclass problem into multiple binary subtasks, whose outputs are combined to obtain the final prediction.

[1]  William W. Cohen.  Fast Effective Rule Induction , 1995, ICML.

[2]  Chih-Jen Lin,et al.  Generalized Bradley-Terry Models and Multi-Class Probability Estimates , 2006, J. Mach. Learn. Res..

[3]  Gunnar Rätsch,et al.  Adapting Codes and Embeddings for Polychotomies , 2002, NIPS.

[4]  I. Kononenko,et al.  INDUCTION OF DECISION TREES USING RELIEFF , 1995 .

[5]  Thomas G. Dietterich,et al.  Solving Multiclass Learning Problems via Error-Correcting Output Codes , 1994, J. Artif. Intell. Res..

[6]  Il-Seok Oh,et al.  Binary classification trees for multi-class classification problems , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[7]  Boonserm Kijsirikul,et al.  Multiclass support vector machines using adaptive directed acyclic graph , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[8]  J. Gama,et al.  A study on Error Correcting Output Codes , 2005, 2005 portuguese conference on artificial intelligence.

[9]  Yoram Singer,et al.  Multiclass Learning by Probabilistic Embeddings , 2002, NIPS.

[10]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  Evolutionary design of multiclass support vector machines , 2007, J. Intell. Fuzzy Syst..

[11]  Eddy Mayoraz,et al.  On the Decomposition of Polychotomies into Dichotomies , 1997, ICML.

[12]  Tong Zhang,et al.  An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods , 2001, AI Mag..

[13]  Chih-Jen Lin,et al.  Probability Estimates for Multi-class Classification by Pairwise Coupling , 2003, J. Mach. Learn. Res..

[14]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[15]  Stefan Kramer,et al.  Ensembles of nested dichotomies for multi-class problems , 2004, ICML.

[16]  Venu Govindaraju,et al.  Half-Against-Half Multi-class Support Vector Machines , 2005, Multiple Classifier Systems.

[17]  Yang Yang,et al.  Fuzzy Multi-class SVM Classifier Based on Optimal Directed Acyclic Graph Using in Similar Handwritten Chinese Characters Recognition , 2005, ISNN.

[18]  Johannes Fürnkranz,et al.  Combining Pairwise Classifiers with Stacking , 2003, IDA.

[19]  Jason Weston,et al.  Support vector machines for multi-class pattern recognition , 1999, ESANN.

[20]  Alon Orlitsky,et al.  On Nearest-Neighbor Error-Correcting Output Codes with Application to All-Pairs Multiclass Support Vector Machines , 2003, J. Mach. Learn. Res..

[21]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[22]  Zhigang Liu,et al.  Hierarchical support vector machines , 2005, IGARSS.

[23]  Johannes Fürnkranz,et al.  Round Robin Classification , 2002, J. Mach. Learn. Res..

[24]  Massimiliano Pontil,et al.  Support Vector Machines for 3D Object Recognition , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  Comparing Techniques for Multiclass Classification Using Binary SVM Predictors , 2004, MICAI.

[26]  Robert Tibshirani,et al.  Classification by Pairwise Coupling , 1997, NIPS.

[27]  Reza Ghaderi,et al.  Coding and decoding strategies for multi-class learning problems , 2003, Inf. Fusion.

[28]  S. Abe,et al.  Decision-tree-based multiclass support vector machines , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[29]  Ulrich H.-G. Kreßel,et al.  Pairwise classification and support vector machines , 1999 .

[30]  Gérard Dreyfus,et al.  Single-layer learning revisited: a stepwise procedure for building and training a neural network , 1989, NATO Neurocomputing.

[31]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[32]  Dorothea Heiss-Czedik,et al.  An Introduction to Genetic Algorithms. , 1997, Artificial Life.

[33]  Wanchai Rivepiboon,et al.  Reordering Adaptive Directed Acyclic Graphs for Multiclass Support Vector Machines , 2003, J. Adv. Comput. Intell. Intell. Informatics.

[34]  Ana Carolina Lorena.  Investigation of strategies for the generation of multiclass support vector machines , 2006, New Challenges in Applied Intelligence Technologies.

[35]  Sergio Escalera,et al.  Decoding of Ternary Error Correcting Output Codes , 2006, CIARP.

[36]  Jennifer G. Dy,et al.  A hierarchical method for multi-class support vector machines , 2004, ICML.

[37]  Günther Palm,et al.  Tree-Structured Support Vector Machines for Multi-class Pattern Recognition , 2001, Multiple Classifier Systems.

[38]  Dwijendra K. Ray-Chaudhuri,et al.  Binary mixture flow with free energy lattice Boltzmann methods , 2022, arXiv.org.

[39]  Jordi Vitrià,et al.  Discriminant ECOC: a heuristic method for application dependent design of error correcting output codes , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Ludmila I. Kuncheva.  Using diversity measures for generating error-correcting output codes in classifier ensembles , 2005, Pattern Recognit. Lett..

[42]  Friedhelm Schwenker,et al.  Hierarchical support vector machines for multi-class pattern recognition , 2000, KES'2000. Fourth International Conference on Knowledge-Based Intelligent Engineering Systems and Allied Technologies. Proceedings (Cat. No.00TH8516).

[43]  Paolo Frasconi,et al.  New results on error correcting output codes of kernel machines , 2004, IEEE Transactions on Neural Networks.

[44]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[45]  Nello Cristianini,et al.  Large Margin DAGs for Multiclass Classification , 1999, NIPS.

[46]  Yoram Singer,et al.  Logistic Regression, AdaBoost and Bregman Distances , 2000, Machine Learning.

[47]  John L. Rhodes,et al.  Algebraic Principles for the Analysis of a Biochemical System , 1967, J. Comput. Syst. Sci..

[48]  Yoram Singer,et al.  Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers , 2000, J. Mach. Learn. Res..

[49]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[50]  Gexiang Zhang,et al.  Automatic Construction Algorithm for Multi-class Support Vector Machines with Binary Tree Architecture , 2006 .

[51]  Boonserm Kijsirikul,et al.  Reordering adaptive directed acyclic graphs: an improved algorithm for multiclass support vector machines , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[52]  Jason Weston,et al.  Multi-Class Support Vector Machines , 1998 .

[53]  Gérard Dreyfus,et al.  Handwritten digit recognition by neural networks with single-layer training , 1992, IEEE Trans. Neural Networks.

[54]  Koby Crammer,et al.  On the Learnability and Design of Output Codes for Multiclass Problems , 2002, Machine Learning.

[55]  Ryan M. Rifkin,et al.  In Defense of One-Vs-All Classification , 2004, J. Mach. Learn. Res..

[56]  S. Sathiya Keerthi,et al.  Which Is the Best Multiclass SVM Method? An Empirical Study , 2005, Multiple Classifier Systems.

[57]  B. Zadrozny.  Reducing multiclass to binary by coupling probability estimates , 2001, NIPS.