Spin discriminant analysis (SDA) - using a one-dimensional classifier for high dimensional classification problems

In this paper we discuss how to use a one-dimensional classifier for solving high dimensional classification problems. We propose Spin Discriminant Analysis (SDA), which enables us to construct a family of new classifiers. We prove that SDA is equivalent to ridged Linear Discriminant Analysis (LDA) when two classes are Gaussians with common covariance matrices. Moreover, we prove that classification based on Parzen's window, is a special case of SDA. In addition to theoretical investigations, we conduct extensive empirical studies, implementing SDA using Support Vector Machines (SVMs) as its one-dimensional classifiers. This SVM-based SDA implementation is named SpinSVM. Our experiments show that SpinSVM outperforms traditional high dimensional classifiers like SVMs, Classification Using Spline (CUS), classification-based Parzen's window, and LDA on most standard and synthetic datasets we tested.

[1]  Calyampudi R. Rao,et al.  Further contributions to the theory of generalized inverse of matrices and its applications , 1971 .

[2]  S. Sain Multivariate locally adaptive density estimation , 2002 .

[3]  M. Kendall Statistical Methods for Research Workers , 1937, Nature.

[4]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[5]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[6]  Sarunas Raudys,et al.  Evolution and generalization of a single neurone: I. Single-layer perceptron as seven statistical classifiers , 1998, Neural Networks.

[7]  Brian D. Ripley,et al.  Neural Networks and Related Methods for Classification , 1994 .

[8]  Paul A. Viola,et al.  Boosting Image Retrieval , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[9]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[10]  Joseph J. Atick,et al.  Towards a Theory of Early Visual Processing , 1990, Neural Computation.

[11]  Smarajit Bose,et al.  Classification using splines , 1996 .

[12]  Jerome H. Friedman,et al.  Flexible Metric Nearest Neighbor Classification , 1994 .

[13]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[14]  G. Wahba Spline models for observational data , 1990 .

[15]  Steven N. MacEachern,et al.  Classification via kernel product estimators , 1998 .

[16]  J. Friedman Special Invited Paper-Additive logistic regression: A statistical view of boosting , 2000 .

[17]  Yoav Freund,et al.  Boosting the margin: A new explanation for the effectiveness of voting methods , 1997, ICML.

[18]  Linda Kaufman,et al.  Solving the quadratic programming problem arising in support vector classification , 1999 .