Overfit prevention in adaptive weighted distance nearest neighbor

Abstract Nearest-Neighbor classification was developed to perform discriminant analysis when reliable parametric estimates of probability densities are unknown or difficult to determine. The major disadvantages of NN are its sensitivity to the distance function and using all training instances in the generalization phase. This can cause slow execution speed and high storage requirement when dealing with large data sets. In our past research, an adaptive distance weighted nearest neighbor algorithm (WDNN) was proposed that tackled both of these problems. WDNN assigns a weight to each training instance that is used in the generalization phase to calculate the distance (or similarity) of a query pattern to that instance. The most disadvantage of WDNN is its early overfit to train data. In the scheme proposed in this paper, we improved WDNN to prevent the overfitting creation and at the same time saved the advantages of the old version of WDNN. For this purpose, the weight of a training instance is updated only if it is effective in the classification of several instances. By this way, the border of classes will be simple and the overfitting is prevented. Experimental results confirm the overfitting prevention.

[1]  Philippe Lauret,et al.  Bayesian neural network approach to short time load forecasting , 2008 .

[2]  Tony R. Martinez,et al.  Reduction Techniques for Instance-Based Learning Algorithms , 2000, Machine Learning.

[3]  John Loughrey,et al.  Using Early-Stopping to Avoid Overfitting in Wrapper-Based Feature Selection Employing Stochastic Search , 2005 .

[4]  Vladimir Cherkassky,et al.  Model complexity control for regression using VC generalization bounds , 1999, IEEE Trans. Neural Networks.

[5]  Hao Wu,et al.  Does overfitting affect performance in estimation of distribution algorithms , 2006, GECCO.

[6]  Robert Ivor John,et al.  A method of learning weighted similarity function to improve the performance of nearest neighbor , 2009, Inf. Sci..

[7]  Enrique Vidal,et al.  Learning weighted metrics to minimize nearest-neighbor classification error , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Tony R. Martinez,et al.  An Integrated Instance‐Based Learning Algorithm , 2000, Comput. Intell..

[9]  Nada Lavrac,et al.  Conditions for Occam's Razor Applicability and Noise Elimination , 1997, ECML.

[10]  Leon N. Cooper,et al.  Improving nearest neighbor rule with a simple adaptive distance measure , 2007, Pattern Recognit. Lett..

[11]  Gerhard Widmer,et al.  Machine Learning: ECML-97 , 1997, Lecture Notes in Computer Science.

[12]  Enrique Vidal,et al.  Learning prototypes and distances (LPD). A prototype reduction technique based on nearest neighbor error minimization , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[13]  M. Pardo,et al.  Learning from data: a tutorial with emphasis on modern pattern recognition methods , 2002 .