Model selection is an important task in machine learning and data mining. When using the holdout testing method to do model selection, a consensus in the machine learning community is that the same model selection goal should be used to identify the best model based on available data. However, following the preliminary work of (Rosset 2004), we show that this is, in general, not true under highly uncertain situations where only very limited data are available. We thoroughly investigate model selection abilities of different measures under highly uncertain situations as we vary model selection goals, learning algorithms and class distributions. The experimental results show that a measure’s model selection ability is relatively stable to the model selection goals and class distributions. However, different learning algorithms call for different measures for model selection. For learning algorithms of SVM and KNN, generally the measures of RMS, SAUC, MXE perform the best. For learning algorithms of decision trees and naive Bayes, generally the measures of RMS, SAUC, MXE, AUC, APR have the best performance.
[1]
V. Vapnik.
Estimation of Dependences Based on Empirical Data
,
2006
.
[2]
Dale Schuurmans.
A New Metric-Based Approach to Model Selection
,
1997,
AAAI/IAAI.
[3]
Catherine Blake,et al.
UCI Repository of machine learning databases
,
1998
.
[4]
Saharon Rosset,et al.
Model selection via the AUC
,
2004,
ICML.
[5]
Ron Kohavi,et al.
The Case against Accuracy Estimation for Comparing Induction Algorithms
,
1998,
ICML.
[6]
Rich Caruana,et al.
Data mining in metric space: an empirical analysis of supervised learning performance criteria
,
2004,
ROCAI.
[7]
Pedro M. Domingos,et al.
Beyond Independence: Conditions for the Optimality of the Simple Bayesian Classifier
,
1996,
ICML.