Population bias control for bagging k-NN experts

We investigate bagging of k - NN classifiers under varying set sizes. For certain set sizes bagging often under-performs due to population bias. We propose a modification to the standard bagging method designed to avoid population bias. The modification leads to substantial performance gains, especially under very small sample size conditions. The choice of the modification method used depends on whether prior knowledge exists or not. If no prior knowledge exists then insuring that all classes exist in the bootstrap set yields the best results.

[1]  Josef Kittler,et al.  Multiple expert system design by combined feature selection and probability level fusion , 2000, Proceedings of the Third International Conference on Information Fusion.

[2]  Robert P. W. Duin,et al.  Bagging for linear classifiers , 1998, Pattern Recognit..

[3]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[4]  Hussein Almuallim,et al.  Turning majority voting classifiers into a single decision tree , 1998, Proceedings Tenth IEEE International Conference on Tools with Artificial Intelligence (Cat. No.98CH36294).

[5]  Tom Heskes.  Balancing Between Bagging and Bumping , 1996, NIPS.

[6]  J. Ross Quinlan,et al.  Bagging, Boosting, and C4.5 , 1996, AAAI/IAAI, Vol. 1.

[7]  Leo Breiman,et al.  Bias, Variance , And Arcing Classifiers , 1996 .

[8]  Robert P. W. Duin,et al.  Stabilizing classifiers for very small sample sizes , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[9]  Jianchang Mao,et al.  A case study on bagging, boosting and basic ensembles of neural networks for OCR , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[10]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[11]  Kamal A. Ali.  A Comparison of Methods for Learning and Combining Evidence From Multiple Models , 1995 .

[12]  David B. Skalak,et al.  Prototype Selection for Composite Nearest Neighbor Classifiers , 1995 .

[13]  David W. Opitz,et al.  An empirical evaluation of bagging and boosting for artificial neural networks , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[14]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..