A Nearest Neighbor Classifier Employing Critical Boundary Vectors for Efficient On-Chip Template Reduction

Aiming at efficient data condensation and improving accuracy, this paper presents a hardware-friendly template reduction (TR) method for the nearest neighbor (NN) classifiers by introducing the concept of critical boundary vectors. A hardware system is also implemented to demonstrate the feasibility of using an field-programmable gate array (FPGA) to accelerate the proposed method. Initially, k-means centers are used as substitutes for the entire template set. Then, to enhance the classification performance, critical boundary vectors are selected by a novel learning algorithm, which is completed within a single iteration. Moreover, to remove noisy boundary vectors that can mislead the classification in a generalized manner, a global categorization scheme has been explored and applied to the algorithm. The global characterization automatically categorizes each classification problem and rapidly selects the boundary vectors according to the nature of the problem. Finally, only critical boundary vectors and k-means centers are used as the new template set for classification. Experimental results for 24 data sets show that the proposed algorithm can effectively reduce the number of template vectors for classification with a high learning speed. At the same time, it improves the accuracy by an average of 2.17% compared with the traditional NN classifiers and also shows greater accuracy than seven other TR methods. We have shown the feasibility of using a proof-of-concept FPGA system of 256 64-D vectors to accelerate the proposed method on hardware. At a 50-MHz clock frequency, the proposed system achieves a 3.86 times higher learning speed than on a 3.4-GHz PC, while consuming only 1% of the power of that used by the PC.

[1]  Amir F. Atiya,et al.  A Novel Template Reduction Approach for the $K$-Nearest Neighbor Method , 2009, IEEE Transactions on Neural Networks.

[2]  Tony R. Martinez,et al.  Reduction Techniques for Instance-Based Learning Algorithms , 2000, Machine Learning.

[3]  Dennis L. Wilson,et al.  Asymptotic Properties of Nearest Neighbor Rules Using Edited Data , 1972, IEEE Trans. Syst. Man Cybern..

[4]  David B. Skalak,et al.  Prototype and Feature Selection by Sampling and Random Mutation Hill Climbing Algorithms , 1994, ICML.

[5]  Jesús Alcalá-Fdez,et al.  KEEL Data-Mining Software Tool: Data Set Repository, Integration of Algorithms and Experimental Analysis Framework , 2011, J. Multiple Valued Log. Soft Comput..

[6]  Noel Lopes,et al.  An Incremental Class Boundary Preserving Hypersphere Classifier , 2011, ICONIP.

[7]  Cor J. Veenman,et al.  The nearest subclass classifier: a compromise between the nearest mean and nearest neighbor classifier , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Annabella Astorino,et al.  Scaling Up Support Vector Machines Using Nearest Neighbor Condensation , 2010, IEEE Transactions on Neural Networks.

[9]  David G. Lowe,et al.  Scalable Nearest Neighbor Algorithms for High Dimensional Data , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Filiberto Pla,et al.  Prototype selection for the nearest neighbour rule through proximity graphs , 1997, Pattern Recognit. Lett..

[11]  José Francisco Martínez Trinidad,et al.  A new fast prototype selection method based on clustering , 2010, Pattern Analysis and Applications.

[12]  Shao-Yi Chien,et al.  Flexible Hardware Architecture of Hierarchical K-Means Clustering for Large Cluster Number , 2011, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[13]  T. Morimoto,et al.  A fully-parallel vector quantization processor for real-time motion picture compression , 1997, 1997 IEEE International Solids-State Circuits Conference. Digest of Technical Papers.

[14]  Carlos Eduardo Pedreira,et al.  Generalized Risk Zone: Selecting Observations for Classification , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Francisco Herrera,et al.  Prototype Selection for Nearest Neighbor Classification: Taxonomy and Empirical Study , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Miguel Toro,et al.  Finding representative patterns with ordered projections , 2003, Pattern Recognit..

[17]  Pedro Larrañaga,et al.  An empirical comparison of four initialization methods for the K-Means algorithm , 1999, Pattern Recognit. Lett..

[18]  Peter E. Hart,et al.  The condensed nearest neighbor rule (Corresp.) , 1968, IEEE Trans. Inf. Theory.

[19]  Elena Marchiori,et al.  Hit Miss Networks with Applications to Instance Selection , 2008, J. Mach. Learn. Res..

[20]  Tadashi Shibata,et al.  Self-adaptive quasi-Gaussian circuits for analog on-chip-trainable multi-class classifiers , 2012, 2012 IEEE International Symposium on Circuits and Systems.

[21]  Cordelia Schmid,et al.  Product Quantization for Nearest Neighbor Search , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Yu Wang,et al.  FPMR: MapReduce framework on FPGA , 2010, FPGA '10.

[23]  Q. Henry Wu,et al.  A class boundary preserving algorithm for data condensation , 2011, Pattern Recognit..

[24]  Ming-Hsuan Yang,et al.  Robust Object Tracking with Online Multiple Instance Learning , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Marco Platzner,et al.  An accelerator for K-TH nearest neighbor thinning based on the IMORC infrastructure , 2009, 2009 International Conference on Field Programmable Logic and Applications.

[26]  Tsutomu Maruyama.  Real-time K-Means Clustering for Color Images on Reconfigurable Hardware , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[27]  Qinghua Hu,et al.  Large-margin nearest neighbor classifiers via sample weight learning , 2011, Neurocomputing.

[28]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[29]  Francesc J. Ferri,et al.  An efficient prototype merging strategy for the condensed 1-NN rule through class-conditional hierarchical clustering , 2002, Pattern Recognit..

[30]  Leon N. Cooper,et al.  Improving nearest neighbor rule with a simple adaptive distance measure , 2006, Pattern Recognit. Lett..

[31]  Olivier Chapelle,et al.  Training a Support Vector Machine in the Primal , 2007, Neural Computation.

[32]  Ta-Wen Kuan,et al.  VLSI Design of an SVM Learning Core on Sequential Minimal Optimization Algorithm , 2012, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[33]  Tadashi Shibata,et al.  Critical Boundary Vector Concept in Nearest Neighbor Classifiers using k-Means Centers for Efficient Template Reduction , 2011, IJCCI.

[34]  Eric Horvitz,et al.  Memory constrained face recognition , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Tadashi Shibata,et al.  An image representation algorithm compatible with neural-associative-processor-based hardware recognition systems , 2003, IEEE Trans. Neural Networks.

[36]  Davide Anguita,et al.  A digital architecture for support vector machines: theory, algorithm, and FPGA implementation , 2003, IEEE Trans. Neural Networks.

[37]  Elena Marchiori,et al.  Class Conditional Nearest Neighbor for Large Margin Instance Selection , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Srihari Cadambi,et al.  A Massively Parallel Digital Learning Processor , 2008, NIPS.

[39]  Sunil Arya,et al.  An optimal algorithm for approximate nearest neighbor searching fixed dimensions , 1998, JACM.

[40]  Azzedine Boukerche,et al.  A Predictive Energy-Efficient Technique to Support Object-Tracking Sensor Networks , 2011, IEEE Transactions on Vehicular Technology.