Kernel Density Classification and Boosting

Kernel density estimation is a commonly used approach to cla ssification. However, most of the theoretical results for kernel methods apply to estimation per seand not necessarily to classification. For example, in estimating the difference between two densities, we show that t e optimal smoothing parameters are increasing functions of the sample size of the complementary group. A re lativ newcomer to the classification portfolio is “boosting”, and this paper proposes an algorithm for boosti ng kernel density classifiers. We note that boosting is closely linked to a previously proposed method of bias reduc tion in kernel density estimation and indicate how it will enjoy similar properties for classification. Numeri cal examples and simulations are used to illustrate the findings, and we also suggest further areas of research. Some key words: Cross-validation; Discrimination; Nonparametric Densit y Estimation; Simulation; Smoothing.

[1]  B. Turlach,et al.  Reducing bias in curve estimation by use of weights , 1999 .

[2]  J. Marron,et al.  Improved Variable Window Kernel Estimates of Probability Densities , 1995 .

[3]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[4]  Yoram Singer,et al.  Improved Boosting Algorithms Using Confidence-rated Predictions , 1998, COLT' 98.

[5]  Charles C. Taylor,et al.  Boosting kernel density estimates: A bias reduction technique? , 2004 .

[6]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[7]  David J. Spiegelhalter,et al.  Machine Learning, Neural and Statistical Classification , 2009 .

[8]  Yoav Freund,et al.  Boosting a weak learning algorithm by majority , 1995, COLT '90.

[9]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[10]  David J. Hand,et al.  Kernel Discriminant Analysis , 1983 .

[11]  J. Habbema,et al.  Cases of doubt in allocation problems , 1974 .

[12]  Matthew P. Wand,et al.  Kernel Smoothing , 1995 .

[13]  Nils Lid Hjort,et al.  On multiplicative bias correction in kernel density estimation , 1999 .

[14]  M. C. Jones,et al.  A Comparison of Higher-Order Bias Kernel Density Estimators , 1997 .

[15]  D. W. Scott,et al.  Variable Kernel Density Estimation , 1992 .

[16]  M. Wand,et al.  On nonparametric discrimination using density differences , 1988 .

[17]  R. Schapire.  The Strength of Weak Learnability , 1990, Machine Learning.

[18]  P. J. Green,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[19]  Jens Perch Nielsen,et al.  A simple bias reduction method for density estimation , 1995 .

[20]  P. Bühlmann,et al.  Boosting with the L2-loss: regression and classification , 2001 .

[21]  M. Hazelton.  Variable kernel density estimation , 2003 .

[22]  Julian Stander,et al.  Nonparametric Density Estimation and Discrimination from Images of Shapes , 1997 .

[23]  B. Yu,et al.  Boosting with the L 2-loss regression and classification , 2001 .

[24]  Ian Abramson.  On Bandwidth Variation in Kernel Estimates-A Square Root Law , 1982 .