Improve regression accuracy by using an attribute weighted KNN approach

KNN (K nearest neighbor) algorithm is a widely used regression method, with a very simple principle about neighborhood. Though it achieves success in many application areas, the method has a shortcoming of weighting equal contributions to each attribute when computing distance between instances. In this paper, we applied a weighted KNN approach by using weights obtained from optimization and feature selection methods and compared the performance and efficiency of these two types of algorithms in regression problems. Experiments on two UCI datasets show that optimization algorithms like particle swarm optimization can obtain more valuable weights than feature selection algorithms, such as information gain and RelefF, with the tradeoff of running time cost. Both of them canimprove the performance of traditional KNN with equal feature weights.

[1]  Francisco Herrera,et al.  Statistical computation of feature weighting schemes through data estimation for nearest neighbor classifiers , 2014, Pattern Recognit..

[2]  Puqiang Zhang,et al.  Data-driven method based on particle swarm optimization and k-nearest neighbor regression for estimating capacity of lithium-ion battery , 2014 .

[3]  José Cristóbal Riquelme Santos,et al.  An evolutionary voting for k-nearest neighbours , 2016, Expert Syst. Appl..

[4]  Thomas Martinetz,et al.  Feature Weighting by Maximum Distance Minimization , 2013, ICANN.

[5]  Yue Shi,et al.  A modified particle swarm optimizer , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[6]  Rory P. Wilson,et al.  Love Thy Neighbour: Automatic Animal Behavioural Classification of Acceleration Data Using the K-Nearest Neighbour Algorithm , 2014, PloS one.

[7]  Igor Kononenko,et al.  Estimating Attributes: Analysis and Extensions of RELIEF , 1994, ECML.

[8]  Ahmed Bouridane,et al.  Simultaneous feature selection and feature weighting using Hybrid Tabu Search/K-nearest neighbor classifier , 2007, Pattern Recognit. Lett..

[9]  Giuseppe De Nicolao,et al.  A new neural network approach for short-term glucose prediction using continuous glucose monitoring time-series and meal information , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[10]  Junheung Park,et al.  Instance variant nearest neighbor using particle swarm optimization for function approximation , 2016, Appl. Soft Comput..

[11]  Jordán Pascual Espada,et al.  Machine learning approach for text and document mining , 2014, ArXiv.

[12]  Sahibsingh A. Dudani.  The Distance-Weighted k-Nearest-Neighbor Rule , 1976, IEEE Transactions on Systems, Man, and Cybernetics.

[13]  Estevam R. Hruschka,et al.  Feature-weighted k-Nearest Neighbor Classifier , 2007, 2007 IEEE Symposium on Foundations of Computational Intelligence.

[14]  D. A. Adeniyi,et al.  Automated web usage data mining and recommendation system using K-Nearest Neighbor (KNN) classification method , 2016 .

[15]  Ahmed Al-Ani,et al.  Optimizing the k-NN metric weights using differential evolution , 2010, 2010 International Conference on Multimedia Computing and Information Technology (MCIT).

[16]  Elena Marchiori,et al.  Class Dependent Feature Weighting and K-Nearest Neighbor Classification , 2013, PRIB.

[17]  Anil K. Jain,et al.  Dimensionality reduction using genetic algorithms , 2000, IEEE Trans. Evol. Comput..

[18]  Zhiping Lin,et al.  Predicting time series with wavelet packet neural networks , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[19]  Mohammad Havaei,et al.  Efficient Interactive Brain Tumor Segmentation as Within-Brain kNN Classification , 2014, 2014 22nd International Conference on Pattern Recognition.

[20]  Jianping Gou,et al.  A new distance-weighted k-nearest neighbor classifier , 2012 .

[21]  Ming Zhu,et al.  Intelligent trading using support vector regression and multilayer perceptrons optimized with genetic algorithms , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).