Classification using distances from samples to linear manifolds

A classifier is proposed wherein the distances from samples to linear manifolds (DSL) are used to perform classification. For each class, a linear manifold is built, whose dimension is high enough to pass all the training samples of the class. The distance from a query sample to a linear manifold is converted to the distance from a point to a linear subspace. And a simple and stable formula is derived to calculate the distance by virtue of the geometrical fundamental of the Gram matrix as well as the regularization technique. The query sample is assigned into the class whose linear manifold is the nearest. On one synthetic data set, thirteen binary-class data sets as well as six multi-class data sets, the experimental results show that the classification performance of DSL is of competence. On most of the data sets, DSL outperforms the comparing classifiers based on k nearest samples or subspaces, and is even superior to support vector machines on some data sets. Further experiment demonstrates that the test efficiency of DSL is also competitive to kNN and the related state-of-the-art classifiers on many data sets.

[1]  Yiguang Liu,et al.  A novel and quick SVM-based multi-class classifier , 2006, Pattern Recognit..

[2]  Shuzhi Sam Ge,et al.  Hand Gesture Recognition and Tracking based on Distributed Locally Linear Embedding , 2006 .

[3]  Josef Kittler,et al.  Moderating k-NN Classifiers , 2002, Pattern Analysis & Applications.

[4]  Alexander J. Smola,et al.  Classification in a normalized feature space using support vector machines , 2003, IEEE Trans. Neural Networks.

[5]  P. Groenen,et al.  Nearest convex hull classification , 2006 .

[6]  Clifford H. Thurber,et al.  Chapter Four – Tikhonov Regularization , 2013 .

[7]  Carl D. Meyer,et al.  Matrix Analysis and Applied Linear Algebra , 2000 .

[8]  Hakan Cevikalp,et al.  Nearest hyperdisk methods for high-dimensional classification , 2008, ICML '08.

[9]  Romulus Cristescu,et al.  Topological Vector Spaces , 1977 .

[10]  Pascal Vincent,et al.  K-Local Hyperplane and Convex Distance Nearest Neighbor Algorithms , 2001, NIPS.

[11]  Dimitrios Gunopulos,et al.  Locally Adaptive Metric Nearest-Neighbor Classification , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Ravi Kothari,et al.  Kernel based subspace pattern classification , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[13]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[14]  Yan Qiu Chen,et al.  The Nearest Neighbor Algorithm of Local Probability Centers , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[15]  Andrew Zisserman,et al.  An Invariant Large Margin Nearest Neighbour Classifier , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[16]  Tingting Mu,et al.  Multiclass Classification Based on Extended Support Vector Data Description , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[17]  Changshui Zhang,et al.  Classification of gene-expression data: The manifold-based metric learning way , 2006, Pattern Recognit..

[18]  José Salvador Sánchez,et al.  On the k-NN performance in a challenging scenario of imbalance and overlapping , 2008, Pattern Analysis and Applications.

[19]  Ricardo H. C. Takahashi,et al.  The $Q$ -Norm Complexity Measure and the Minimum Gradient Method: A Novel Approach to the Machine Learning Structural Risk Minimization Problem , 2008, IEEE Transactions on Neural Networks.

[20]  Amir F. Atiya,et al.  A Novel Template Reduction Approach for the $K$-Nearest Neighbor Method , 2009, IEEE Transactions on Neural Networks.

[21]  George Kollios,et al.  BoostMap: An Embedding Method for Efficient Nearest Neighbor Retrieval , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Chih-Jen Lin,et al.  Asymptotic Behaviors of Support Vector Machines with Gaussian Kernel , 2003, Neural Computation.

[23]  J. L. Hodges,et al.  Discriminatory Analysis - Nonparametric Discrimination: Consistency Properties , 1989 .

[24]  Olvi L. Mangasarian,et al.  Multisurface proximal support vector machine classification via generalized eigenvalues , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Jing Peng,et al.  Kernel Pooled Local Subspaces for Classification , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[26]  Hakan Cevikalp,et al.  Manifold Based Local Classifiers: Linear and Nonlinear Approaches , 2010, J. Signal Process. Syst..

[27]  Robert Tibshirani,et al.  Discriminant Adaptive Nearest Neighbor Classification , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Leon N. Cooper,et al.  Neighborhood size selection in the k-nearest-neighbor rule using statistical confidence , 2006, Pattern Recognit..

[29]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[30]  A. Atiya,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2005, IEEE Transactions on Neural Networks.

[31]  Nicolás García-Pedrajas,et al.  Constructing Ensembles of Classifiers by Means of Weighted Instance Selection , 2009, IEEE Transactions on Neural Networks.

[32]  Hanan Samet,et al.  K-Nearest Neighbor Finding Using MaxNearestDist , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Deyu Meng,et al.  Fast and Efficient Strategies for Model Selection of Gaussian Support Vector Machine , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[34]  José Martínez Sotoca,et al.  An analysis of how training data complexity affects the nearest neighbor classifiers , 2007, Pattern Analysis and Applications.

[35]  Liang Wang,et al.  Learning and Matching of Dynamic Shape Manifolds for Human Action Recognition , 2007, IEEE Transactions on Image Processing.

[36]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[37]  Yann LeCun,et al.  Transformation Invariance in Pattern Recognition-Tangent Distance and Tangent Propagation , 1996, Neural Networks: Tricks of the Trade.

[38]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[39]  David Zhang,et al.  On kernel difference-weighted k-nearest neighbor classification , 2008, Pattern Analysis and Applications.