Bayes Error Estimation Using Parzen and k-NN Procedures

The use of k nearest neighbor (k-NN) and Parzen density estimates to obtain estimates of the Bayes error is investigated under limited design set conditions. By drawing analogies between the k-NN and Parzen procedures, new procedures are suggested, and experimental results are given which indicate that these procedures yield a significant improvement over the conventional k-NN and Parzen procedures. We show that, by varying the decision threshold, many of the biases associated with the k-NN or Parzen density estimates may be compensated, and successful error estimation may be performed in spite of these biases. Experimental results are given which demonstrate the effect of kernel size and shape (Parzen), the size of k (k-NN), and the number of samples in the design set.

[1]  M. Rosenblatt.  Remarks on Some Nonparametric Estimates of a Density Function , 1956 .

[2]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[3]  C. Quesenberry,et al.  A nonparametric estimate of a multivariate density function , 1965 .

[4]  T. Cacoullos.  Estimation of a multivariate density , 1966 .

[5]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[6]  Larry D. Hostetler,et al.  Optimization of k nearest neighbor density estimates , 1973, IEEE Trans. Inf. Theory.

[7]  J. Habbema.  A stepwise discriminant analysis program using density estimetion , 1974 .

[8]  T. Wagner,et al.  Nonparametric estimates of probability densities , 1975, IEEE Trans. Inf. Theory.

[9]  Robert P. W. Duin,et al.  On the Choice of Smoothing Parameters for Parzen Estimators of Probability Density Functions , 1976, IEEE Transactions on Computers.

[10]  C. J. Stone,et al.  Consistent Nonparametric Regression , 1977 .

[11]  J. Yackel,et al.  Consistency Properties of Nearest Neighbor Density Function Estimators , 1977 .

[12]  L. Devroye,et al.  The Strong Uniform Consistency of Nearest Neighbor Density Estimates. , 1977 .

[13]  B. Silverman,et al.  Choosing the window width when estimating a density , 1978 .

[14]  Anil K. Jain,et al.  ON BALANCING DECISION FUNCTIONS. , 1979 .

[15]  John Van Ness,et al.  On the dominance of non-parametric Bayes rule discriminant algorithms in high dimensions , 1980, Pattern Recognit..

[16]  Luc Devroye,et al.  On the Inequality of Cover and Hart in Nearest Neighbor Discrimination , 1981, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  L. Devroye.  On the Almost Everywhere Convergence of Nonparametric Regression Function Estimates , 1981 .

[18]  Keinosuke Fukunaga,et al.  A Nonparametric Two-Dimensional Display for Classification , 1982, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  L. Devroye.  Necessary and sufficient conditions for the pointwise convergence of nearest neighbor regression function estimates , 1982 .

[20]  David J. Hand,et al.  Kernel Discriminant Analysis , 1983 .

[21]  P. Hall.  Large Sample Optimality of Least Squares Cross-Validation in Density Estimation , 1983 .

[22]  Keinosuke Fukunaga,et al.  Classification Error for a Very Large Number of Classes , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Keinosuke Fukunaga,et al.  Bias of Nearest Neighbor Error Estimates , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  J. L. Hodges,et al.  Discriminatory Analysis - Nonparametric Discrimination: Consistency Properties , 1989 .

[25]  S.,et al.  CONSISTENT CROSS-VALIDATED DENSITY ESTIMATION , 2022 .