For two-class discrimination, Xie and Qiu [The effect of imbalanced data sets on LDA: a theoretical and empirical analysis, Pattern Recognition 40 (2) (2007) 557-562] claimed that, when covariance matrices of the two classes were unequal, a (class) unbalanced data set had a negative effect on the performance of linear discriminant analysis (LDA). Through re-balancing 10 real-world data sets, Xie and Qiu [The effect of imbalanced data sets on LDA: a theoretical and empirical analysis, Pattern Recognition 40 (2) (2007) 557-562] provided empirical evidence to support the claim using AUC (Area Under the receiver operating characteristic Curve) as the performance metric. We suggest that such a claim is vague if not misleading, there is no solid theoretical analysis presented in Xie and Qiu [The effect of imbalanced data sets on LDA: a theoretical and empirical analysis, Pattern Recognition 40 (2) (2007) 557-562], and AUC can lead to a quite different conclusion from that led to by misclassification error rate (ER) on the discrimination performance of LDA for unbalanced data sets. Our empirical and simulation studies suggest that, for LDA, the increase of the median of AUC (and thus the improvement of performance of LDA) from re-balancing is relatively small, while, in contrast, the increase of the median of ER (and thus the decline in performance of LDA) from re-balancing is relatively large. Therefore, from our study, there is no reliable empirical evidence to support the claim that a (class) unbalanced data set has a negative effect on the performance of LDA. In addition, re-balancing affects the performance of LDA for data sets with either equal or unequal covariance matrices, indicating that having unequal covariance matrices is not a key reason for the difference in performance between original and re-balanced data.
[1]
Foster J. Provost,et al.
Learning When Training Data are Costly: The Effect of Class Distribution on Tree Induction
,
2003,
J. Artif. Intell. Res..
[2]
E. S. Gilbert,et al.
The effect of unequal variance-covariance matrices on Fisher's linear discriminant function.
,
1969,
Biometrics.
[3]
Andrew P. Bradley,et al.
The use of the area under the ROC curve in the evaluation of machine learning algorithms
,
1997,
Pattern Recognit..
[4]
G. McLachlan.
Discriminant Analysis and Statistical Pattern Recognition
,
1992
.
[5]
D. J. Newman,et al.
UCI Repository of Machine Learning Database
,
1998
.
[6]
Gary M. Weiss.
Mining with rarity: a unifying framework
,
2004,
SKDD.
[7]
J. Hanley,et al.
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
,
1982,
Radiology.
[8]
Nitesh V. Chawla,et al.
Editorial: special issue on learning from imbalanced data sets
,
2004,
SKDD.
[9]
Zhengding Qiu,et al.
The effect of imbalanced data sets on LDA: A theoretical and empirical analysis
,
2007,
Pattern Recognit..
[10]
Catherine Blake,et al.
UCI Repository of machine learning databases
,
1998
.
[11]
Mehryar Mohri,et al.
AUC Optimization vs. Error Rate Minimization
,
2003,
NIPS.
[12]
T. W. Anderson,et al.
Classification into two Multivariate Normal Distributions with Different Covariance Matrices
,
1962
.
[13]
Michael I. Jordan,et al.
On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes
,
2001,
NIPS.
[14]
Sidney Marks,et al.
Discriminant Functions When Covariance Matrices are Unequal
,
1974
.
[15]
D. Titterington,et al.
Comparison of Discrimination Techniques Applied to a Complex Data Set of Head Injured Patients
,
1981
.