Bagging and the Random Subspace Method for Redundant Feature Spaces

The performance of a single weak classifier can be improved by using combining techniques such as bagging, boosting and the random subspace method. When applying them to linear discriminant analysis, it appears that they are useful in different situations. Their performance is strongly affected by the choice of the base classifier and the training sample size. As well, their usefulness depends on the data distribution. In this paper, on the example of the pseudo Fisher linear classifier, we study the effect of the redundancy in the data feature set on the performance of the random subspace method and bagging.

[1]  Robert P. W. Duin,et al.  Boosting in Linear Discriminant Analysis , 2000, Multiple Classifier Systems.

[2]  Robert P. W. Duin,et al.  Bagging for linear classifiers , 1998, Pattern Recognit..

[3]  L. D. Baumert,et al.  The Search for Hadamard Matrices , 1963 .

[4]  Keinosuke Fukunaga,et al.  Introduction to statistical pattern recognition (2nd ed.) , 1990 .

[5]  Guozhong An,et al.  The Effects of Adding Noise During Backpropagation Training on a Generalization Performance , 1996, Neural Computation.

[6]  Tin Kam Ho,et al.  Complexity of Classification Problems and Comparative Advantages of Combined Classifiers , 2000, Multiple Classifier Systems.

[7]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[8]  J. Friedman.  Regularized Discriminant Analysis , 1989 .

[9]  Dov Dori,et al.  Advances in Pattern Recognition , 1998, Lecture Notes in Computer Science.

[10]  Robert P. W. Duin,et al.  Bagging, Boosting and the Random Subspace Method for Linear Classifiers , 2002, Pattern Analysis & Applications.

[11]  Robert P. W. Duin,et al.  The Role of Combining Rules in Bagging and Boosting , 2000, SSPR/SPR.

[12]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[14]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[15]  Anil K. Jain,et al.  39 Dimensionality and sample size considerations in pattern recognition practice , 1982, Classification, Pattern Recognition and Reduction of Dimensionality.