Boosting SVM Classifiers with Logistic Regression

The support vector machine classifier is a linear maximum margin classifier. It performs very well in many classification applications. Although, it could be extended to nonlinear cases by exploiting the idea of kernel, it might still suffer from the heterogeneity in the training examples. Since there are very few theories in the literature to guide us on how to choose kernel functions, the selection of kernel is usually based on a try-and-error manner. When the training set are imbalanced, the data set might not be linear separable in the feature space defined by the chosen kernel. In this paper, we propose a hybrid method by integrating “small” support vector machine classifiers by logistic regression models. By appropriately partitioning the training set, this ensemble classifier can improve the performance of the SVM classifier trained with whole training examples at a time. With this method, we can not only avoid the difficulty of the heterogeneity, but also have probability outputs for all examples. Moreover, it is less ambiguous than the classifiers combined with voting schemes. From our simulation studies and some empirical results, we find that these kinds of hybrid SVM classifiers are robust in the following sense: (1) It improves the performance (prediction accuracy) of the SVM classifier trained with whole training set, when there are some kind of heterogeneity in the training examples; and (2) it is at least as good as the original SVM classifier, when there is actually no heterogeneity presented in the training examples. We also apply this hybrid method to multi-class problems by replacing binary logistic regression models with polychotomous logistic regression models. Moreover, the polychotomous regression model can be constructed from individual binary logistic regression models.