On the Relationship Between the Support Vector Machine for Classification and Sparsified Fisher's Linear Discriminant

We show that the orientation and location of the separating hyperplane for 2-class supervised pattern classification obtained by the Support Vector Machine (SVM) proposed by Vapnik and his colleagues, is equivalent to the solution obtained by Fisher's Linear Discriminant on the set of Support Vectors. In other words, SVM can be seen as a way to ‘sparsify’ Fisher's Linear Discriminant in order to obtain the most generalizing classification from the training set.

[1]  Massimiliano Pontil,et al.  Recognizing 3-D Objects with Linear Support Vector Machines , 1998, ECCV.

[2]  R. Courant,et al.  Methods of Mathematical Physics , 1962 .

[3]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[4]  Daphna Weinshall,et al.  Condensing image databases when retrieval is based on non-metric distances , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[5]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[6]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[7]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[8]  R W Prager,et al.  Development of low entropy coding in a recurrent network. , 1996, Network.

[9]  A. Tversky.  Features of Similarity , 1977 .

[10]  Federico Girosi,et al.  An Equivalence Between Sparse Approximation and Support Vector Machines , 1998, Neural Computation.

[11]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[12]  Harry V. Roberts,et al.  Nature statistics , 1989, Nature.

[13]  Vladimir Vapnik,et al.  The Nature of Statistical Learning , 1995 .

[14]  Federico Girosi,et al.  Training support vector machines: an application to face detection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[16]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[17]  Azriel Rosenfeld,et al.  Robust regression methods for computer vision: A review , 1991, International Journal of Computer Vision.