An Empirical Evaluation of Supervised Learning for ROC Area

We present an empirical comparison of the AUC performance of seven supervised learning methods: SVMs, neural nets, decision trees, k-nearest neighbor, bagged trees, boosted trees, and boosted stumps. Overall, boosted trees have the best average AUC performance, followed by bagged trees, neural nets and SVMs. We then present an ensemble selection method that yields even better AUC. Ensembles are built with forward stepwise selection, the model that maximizes ensemble AUC performance being added at each step. The proposed method builds ensembles that outperform the best individual model on all the seven test problems.

[1]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[2]  Tom Fawcett,et al.  Analysis and Visualization of Classifier Performance: Comparison under Imprecise Class and Cost Distributions , 1997, KDD.

[3]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[4]  Robert E. Schapire,et al.  The Boosting Approach to Machine Learning An Overview , 2003 .

[5]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[6]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[7]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[8]  Constantin F. Aliferis,et al.  An evaluation of machine-learning methods for predicting pneumonia mortality , 1997, Artif. Intell. Medicine.

[9]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.

[10]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[11]  Thomas G. Dietterich,et al.  Solving Multiclass Learning Problems via Error-Correcting Output Codes , 1994, J. Artif. Intell. Res..

[12]  Pedro M. Domingos.  Bayesian Averaging of Classifiers and the Overfitting Problem , 2000, ICML.

[13]  OpitzDavid,et al.  Popular ensemble methods , 1999 .

[14]  David W. Opitz,et al.  Feature Selection for Ensembles , 1999, AAAI/IAAI.

[15]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[16]  Paul W. Munro,et al.  Competition Among Networks Improves Committee Performance , 1996, NIPS.

[17]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[18]  Harris Drucker,et al.  Comparison of learning algorithms for handwritten digit recognition , 1995 .

[19]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[20]  Robert F. Cromp,et al.  Support Vector Machine Classifiers as Applied to AVIRIS Data , 1999 .