Efficient Decision Trees for Multi–Class Support Vector Machines Using Entropy and Generalization Error Estimation

Abstract We propose new methods for support vector machines using a tree architecture for multi-class classification. In each node of the tree, we select an appropriate binary classifier, using entropy and generalization error estimation, then group the examples into positive and negative classes based on the selected classifier, and train a new classifier for use in the classification phase. The proposed methods can work in time complexity between O(log2 N) and O(N), where N is the number of classes. We compare the performance of our methods with traditional techniques on the UCI machine learning repository using 10-fold cross-validation. The experimental results show that the methods are very useful for problems that need fast classification time or those with a large number of classes, since the proposed methods run much faster than the traditional techniques but still provide comparable accuracy.

[1]  Cheng Wang,et al.  Adaptive binary tree for fast SVM multiclass classification , 2009, Neurocomputing.

[2]  Francisco Herrera,et al.  Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power , 2010, Inf. Sci..

[3]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[4]  Boonserm Kijsirikul,et al.  Multiclass support vector machines using adaptive directed acyclic graph , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[5]  Vladimir Vapnik,et al.  An overview of statistical learning theory , 1999, IEEE Trans. Neural Networks.

[6]  B. Fei,et al.  Binary tree of SVM: a new fast multiclass training and classification algorithm , 2006, IEEE Transactions on Neural Networks.

[7]  John Shawe-Taylor,et al.  Generalization Performance of Support Vector Machines and Other Pattern Classifiers , 1999 .

[8]  Xiaowei Yang,et al.  The one-against-all partition based binary tree support vector machine algorithms for multi-class classification , 2013, Neurocomputing.

[9]  Koby Crammer,et al.  On the Learnability and Design of Output Codes for Multiclass Problems , 2002, Machine Learning.

[10]  Boonserm Kijsirikul,et al.  Information-based dichotomization: A method for multiclass Support Vector Machines , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[11]  Venu Govindaraju,et al.  Half-Against-Half Multi-class Support Vector Machines , 2005, Multiple Classifier Systems.

[12]  Gérard Dreyfus,et al.  Single-layer learning revisited: a stepwise procedure for building and training a neural network , 1989, NATO Neurocomputing.

[13]  Soo-Young Lee,et al.  Support Vector Machines with Binary Tree Architecture for Multi-Class Classification , 2004 .

[14]  Kristin P. Bennett,et al.  Multicategory Classification by Support Vector Machines , 1999, Comput. Optim. Appl..

[15]  Jinglu Hu,et al.  A hierarchical SVM based multiclass classification by using similarity clustering , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[16]  Dejan Gjorgjevikj,et al.  A Multi-class SVM Classifier Utilizing Binary Decision Tree , 2009, Informatica.

[17]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[18]  S. Abe,et al.  Decision-tree-based multiclass support vector machines , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[19]  Robert Tibshirani,et al.  Classification by Pairwise Coupling , 1997, NIPS.

[20]  Boonserm Kijsirikul,et al.  Enhancement of multi-class support vector machine construction from binary learners using generalization performance , 2013, Neurocomputing.

[21]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[22]  R. K. Agrawal,et al.  Optimal Decision Tree Based Multi-class Support Vector Machine , 2011, Informatica.

[23]  Nello Cristianini,et al.  Large Margin DAGs for Multiclass Classification , 1999, NIPS.

[24]  Philip S. Yu,et al.  Multi-Space-Mapped SVMs for Multi-class Classification , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[25]  Madan Gopal,et al.  Reduced one-against-all method for multiclass SVM classification , 2011, Expert Syst. Appl..

[26]  Madan Gopal,et al.  Fast Multiclass SVM Classification Using Decision Tree Based One-Against-All Method , 2010, Neural Processing Letters.

[27]  Katarzyna Stapor,et al.  Using the one–versus–rest strategy with samples balancing to improve pairwise coupling classification , 2016, Int. J. Appl. Math. Comput. Sci..

[28]  Shigeo Abe,et al.  Multiclass Support Vector Machines , 2010 .

[29]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[30]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .