Consistency and convergence rate for nearest subspace classifier

The Nearest subspace classifier (NSS) finds an estimation of the underlying subspace within each class and assigns data points to the class that corresponds to its nearest subspace. This paper mainly studies how well NSS can be generalized to new samples. It is proved that NSS is strongly consistent and has rate of convergence O(n−1/2) under certain assumptions. Some simulations are presented eventually to verify the theoretical results.

[1]  Pascal Vincent,et al.  K-Local Hyperplane and Convex Distance Nearest Neighbor Algorithms , 2001, NIPS.

[2]  Lawrence Sirovich,et al.  Application of the Karhunen-Loeve Procedure for the Characterization of Human Faces , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Wladyslaw Skarbek,et al.  Local Subspace Method for Pattern Recognition , 1997, CAIP.

[4]  Nathan Srebro,et al.  Fast Rates for Regularized Objectives , 2008, NIPS.

[5]  R. Girard.  Fast rate of convergence in high-dimensional linear discriminant analysis , 2009, 0909.2191.

[6]  Takeo Kanade,et al.  A Multibody Factorization Method for Independently Moving Objects , 1998, International Journal of Computer Vision.

[7]  L. Gyorfi.  The rate of convergence of k_n -NN regression estimates and classification rules (Corresp.) , 1981 .

[8]  Dario Maio,et al.  Multispace KL for Pattern Representation and Classification , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[10]  Ingo Steinwart,et al.  Support Vector Machines are Universally Consistent , 2002, J. Complex..

[11]  Jean-Philippe Vert,et al.  Consistency and Convergence Rates of One-Class SVMs and Related Algorithms , 2006, J. Mach. Learn. Res..

[12]  Koji Tsuda.  Subspace classifier in reproducing kernel Hilbert space , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[13]  David Lindley,et al.  Advanced Statistical Methods in Biometric Research. , 1953 .

[14]  L. Devroye,et al.  Nonparametric density estimation : the L[1] view , 1987 .

[15]  Wei Liang,et al.  A Graph Based Subspace Semi-supervised Learning Framework for Dimensionality Reduction , 2008, ECCV.

[16]  Sanjoy Dasgupta,et al.  Rates of Convergence for Nearest Neighbor Classification , 2014, NIPS.

[17]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[18]  Russell A. Epstein,et al.  5/spl plusmn/2 eigenimages suffice: an empirical investigation of low-dimensional lighting models , 1995, Proceedings of the Workshop on Physics-Based Modeling in Computer Vision.

[19]  László Györfi,et al.  The Rate of Convergence of k ,-NN Regression Estimates and Classification Rules , 1978 .

[20]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[21]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[22]  Fatih Murat Porikli,et al.  Connecting the dots in multi-class classification: From nearest subspace to collaborative representation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Ingo Steinwart,et al.  Fast rates for support vector machines using Gaussian kernels , 2007, 0708.1838.

[24]  J. R. Quinlan.  Discovering rules by induction from large collections of examples Intro-ductory readings in expert s , 1979 .

[25]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[26]  Erkki Oja,et al.  Subspace methods of pattern recognition , 1983 .

[27]  Dongfang Zou.  Local Subspace Classifier in Reproducing Kernel Hilbert Space , 2000, ICMI.

[28]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[29]  W. Zhao.  Subspace methods in object/face recognition , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[30]  Guillermo Sapiro,et al.  Discriminative k-metrics , 2009, ICML '09.

[31]  Heikki Riittinen,et al.  Spectral classification of phonemes by learning subspaces , 1979, ICASSP.

[32]  Gilles Blanchard,et al.  On the Rate of Convergence of Regularized Boosting Classifiers , 2003, J. Mach. Learn. Res..

[33]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[34]  J. Franklin,et al.  The elements of statistical learning: data mining, inference and prediction , 2005 .

[35]  Ronen Basri,et al.  Lambertian reflectance and linear subspaces , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[36]  Hiroshi Murase,et al.  Kernel-Based Nonlinear Subspace Method for Pattern Recognition , 2002, Systems and Computers in Japan.

[37]  Tong Zhang.  Statistical behavior and consistency of classification methods based on convex risk minimization , 2003 .

[38]  C. J. Stone,et al.  Consistent Nonparametric Regression , 1977 .

[39]  Peter L. Bartlett,et al.  AdaBoost is Consistent , 2006, J. Mach. Learn. Res..

[40]  Hakan Cevikalp,et al.  Manifold Based Local Classifiers: Linear and Nonlinear Approaches , 2010, J. Signal Process. Syst..

[41]  Gilles Blanchard,et al.  On the Convergence of Eigenspaces in Kernel Principal Component Analysis , 2005, NIPS.

[42]  Infotech Oulu,et al.  Protein Fold Recognition with K-Local Hyperplane Distance Nearest Neighbor Algorithm , 2004 .

[43]  P. Massart,et al.  Statistical performance of support vector machines , 2008, 0804.0551.

[44]  Adolfo Hernández,et al.  On the consistency properties of linear and quadratic discriminant analyses , 2005 .

[45]  L. Devroye,et al.  Nonparametric Density Estimation: The L 1 View. , 1985 .

[46]  Ravi Kothari,et al.  Kernel based subspace pattern classification , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[47]  David J. Kriegman,et al.  Clustering appearances of objects under varying illumination conditions , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[48]  Shuzhi Sam Ge,et al.  $k$-NS: A Classifier by the Distance to the Nearest Subspace , 2011, IEEE Transactions on Neural Networks.

[49]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[50]  Calyampudi Radhakrishna Rao,et al.  Advanced Statistical Methods in Biometric Research. , 1953 .

[51]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[52]  Michael I. Jordan,et al.  Convexity, Classification, and Risk Bounds , 2006 .

[53]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[54]  W. Kahan,et al.  The Rotation of Eigenvectors by a Perturbation. III , 1970 .

[55]  J. L. Hodges,et al.  Discriminatory Analysis - Nonparametric Discrimination: Consistency Properties , 1989 .

[56]  Stanislav Minsker.  On Some Extensions of Bernstein's Inequality for Self-adjoint Operators , 2011, 1112.5448.

[57]  Bernhard Schölkopf,et al.  Extracting Support Data for a Given Task , 1995, KDD.

[58]  Vladimir Vapnik,et al.  An overview of statistical learning theory , 1999, IEEE Trans. Neural Networks.

[59]  Sanjeev R. Kulkarni,et al.  Rates of convergence of nearest neighbor estimation under arbitrary sampling , 1995, IEEE Trans. Inf. Theory.

[60]  Ding-Xuan Zhou,et al.  SVM Soft Margin Classifiers: Linear Programming versus Quadratic Programming , 2005, Neural Computation.

[61]  Chandler Davis.  The rotation of eigenvectors by a perturbation , 1963 .

[62]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[63]  Yiming Ying,et al.  Support Vector Machine Soft Margin Classifiers: Error Analysis , 2004, J. Mach. Learn. Res..

[64]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .