k-relevance vectors: Considering relevancy beside nearness

Abstract This study combines two different learning paradigms, k-nearest neighbor (k-NN) rule, as memory-based learning paradigm and relevance vector machines (RVM), as statistical learning paradigm. The purpose is to improve the performance of k-NN rule through selection of important features with sparse Bayesian learning method. This combination is performed in kernel space and is called k-relevance vector (k-RV). The proposed model significantly prunes irrelevant features. Our combination of k-NN and RVM presents a new concept of similarity measurement for k-NN rule, we call it k-relevancy which aims to consider “relevancy” in the feature space beside “nearness” in the input space. We also introduce a new parameter, responsible for early stopping of iterations in RVM that is able to improve the classification accuracy. Intensive experiments are conducted on several classification datasets from University of California Irvine (UCI) repository and two real datasets from computer vision domain. The performance of k-RV is highly competitive compared to a few state-of-the-arts in terms of classification accuracy.

[1]  Chi-Man Vong,et al.  Sparse Bayesian Extreme Learning Machine for Multi-classification , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[2]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[3]  B. Gopinath,et al.  Development of an Automated Medical Diagnosis System for Classifying Thyroid Tumor Cells using Multiple Classifier Fusion , 2015, Technology in cancer research & treatment.

[4]  Gürsel Serpen,et al.  Host-based misuse intrusion detection using PCA feature extraction and kNN classification algorithms , 2018, Intell. Data Anal..

[5]  Euntai Kim,et al.  Proposing a GPU based modified fuzzy nearest neighbor rule for traffic sign detection , 2015, 2015 15th International Conference on Control, Automation and Systems (ICCAS).

[6]  Chih-Fong Tsai,et al.  CANN: An intrusion detection system based on combining cluster centers and nearest neighbors , 2015, Knowl. Based Syst..

[7]  Francisco Herrera,et al.  kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data , 2017, Knowl. Based Syst..

[8]  David J. C. MacKay,et al.  The Evidence Framework Applied to Classification Networks , 1992, Neural Computation.

[9]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[10]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[11]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[12]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[13]  Gongde Guo,et al.  Nearest neighbor classification of categorical data by attributes weighting , 2015, Expert Syst. Appl..

[14]  Lin Zhu,et al.  A graph-based semi-supervised k nearest-neighbor method for nonlinear manifold distributed data classification , 2016, Inf. Sci..

[15]  Kevin A. Schneider,et al.  Breast Cancer Diagnosis with Transfer Learning and Global Pooling , 2019, 2019 International Conference on Information and Communication Technology Convergence (ICTC).

[16]  Robert Ivor John,et al.  A method of learning weighted similarity function to improve the performance of nearest neighbor , 2009, Inf. Sci..

[17]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[18]  Ying Ma,et al.  A hybrid method based on extreme learning machine and k-nearest neighbor for cloud classification of ground-based visible cloud image , 2015, Neurocomputing.

[19]  Taeho Jo,et al.  Using K Nearest Neighbors for text segmentation with feature similarity , 2017, 2017 International Conference on Communication, Control, Computing and Electronics Engineering (ICCCCEE).

[20]  Kevin A. Schneider,et al.  Automatic Detection of Coronavirus Disease (COVID-19) in X-ray and CT Images: A Machine Learning Based Approach , 2020, Biocybernetics and Biomedical Engineering.

[21]  Peyman Hosseinzadeh Kassani,et al.  A comparative study of deep learning architectures on melanoma detection. , 2019, Tissue & cell.

[22]  Johannes Stallkamp,et al.  The German Traffic Sign Recognition Benchmark: A multi-class classification competition , 2011, The 2011 International Joint Conference on Neural Networks.

[23]  Masashi Sugiyama,et al.  Target Neighbor Consistent Feature Weighting for Nearest Neighbor Classification , 2011, NIPS.

[24]  Philip S. Yu,et al.  Top 10 algorithms in data mining , 2007, Knowledge and Information Systems.

[25]  Sara Hosseinzadeh Kassani,et al.  Introducing a hybrid model of DEA and data mining in evaluating efficiency. Case study: Bank Branches , 2018, ArXiv.

[26]  D. N. Tibarewala,et al.  Classification of lower limb motor imagery using K Nearest Neighbor and Naïve-Bayesian classifier , 2016, 2016 3rd International Conference on Recent Advances in Information Technology (RAIT).

[27]  J. Ross Quinlan,et al.  Simplifying Decision Trees , 1987, Int. J. Man Mach. Stud..

[28]  Andrew Beng Jin Teoh,et al.  A new sparse model for traffic sign classification using soft histogram of oriented gradients , 2017, Appl. Soft Comput..

[29]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[30]  Michael E. Tipping.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001, J. Mach. Learn. Res..

[31]  Nada Lavrac,et al.  Selected techniques for data mining in medicine , 1999, Artif. Intell. Medicine.

[32]  Carl E. Rasmussen,et al.  Healing the relevance vector machine through augmentation , 2005, ICML.

[33]  Mirko Perkusich,et al.  Early diagnosis of gastrointestinal cancer by using case-based and rule-based reasoning , 2016, Expert Syst. Appl..

[34]  Lorenzo Rosasco,et al.  Nonparametric sparsity and regularization , 2012, J. Mach. Learn. Res..

[35]  David Zhang,et al.  On kernel difference-weighted k-nearest neighbor classification , 2008, Pattern Analysis and Applications.

[36]  Gerhard Tutz,et al.  Improved methods for the imputation of missing data by nearest neighbor methods , 2015, Comput. Stat. Data Anal..

[37]  Prachi Surlakar,et al.  Comparative Analysis of K-Means and K-Nearest Neighbor Image Segmentation Techniques , 2016, 2016 IEEE 6th International Conference on Advanced Computing (IACC).

[38]  Julio López,et al.  Redefining nearest neighbor classification in high-dimensional settings , 2018, Pattern Recognit. Lett..

[39]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[40]  Andrew Beng Jin Teoh,et al.  Online Heterogeneous Face Recognition Based on Total-Error-Rate Minimization , 2020, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[41]  Kar-Ann Toh,et al.  Benchmarking a reduced multivariate polynomial pattern classifier , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.