Large Margin Classifier Based on Affine Hulls

This paper introduces a geometrically inspired large-margin classifier that can be a better alternative to the Support Vector Machines (SVMs) for the classification problems with limited number of training samples. In contrast to the SVM classifier, we approximate classes with affine hulls of their class samples rather than convex hulls, which may be unrealistically tight in high-dimensional spaces. To find the best separating hyperplane between any pair of classes approximated with the affine hulls, we first compute the closest points on the affine hulls and connect these two points with a line segment. The optimal separating hyperplane is chosen to be the hyperplane that is orthogonal to the line segment and bisects the line. To allow soft margin solutions, we first reduce affine hulls in order to alleviate the effects of outliers and then search for the best separating hyperplane between these reduced models. Multi-class classification problems are dealt with constructing and combining several binary classifiers as in SVM. The experiments on several databases show that the proposed method compares favorably with the SVM classifier.

[1]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[2]  Kristin P. Bennett,et al.  Duality and Geometry in SVM Classifiers , 2000, ICML.

[3]  Jennifer G. Dy,et al.  A hierarchical method for multi-class support vector machines , 2004, ICML.

[4]  Takeo Kanade,et al.  Robust L/sub 1/ norm factorization in the presence of outliers and missing data by alternative convex programming , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[5]  Bernhard Schölkopf,et al.  New Support Vector Algorithms , 2000, Neural Computation.

[6]  Aleix M. Martinez,et al.  The AR face database , 1998 .

[7]  M. Bilginer Gülmezoglu,et al.  The common vector approach and its relation to principal component analysis , 2001, IEEE Trans. Speech Audio Process..

[8]  Jakob J. Verbeek,et al.  Learning nonlinear image manifolds by global alignment of local linear models , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Hakan Cevikalp,et al.  Manifold Based Local Classifiers: Linear and Nonlinear Approaches , 2010, J. Signal Process. Syst..

[10]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[11]  A. Roli.  Artificial Neural Networks , 2012, Lecture Notes in Computer Science.

[12]  Pascal Vincent,et al.  K-Local Hyperplane and Convex Distance Nearest Neighbor Algorithms , 2001, NIPS.

[13]  Chih-Jen Lin,et al.  Manuscript Number: 2187 Training ν-Support Vector Classifiers: Theory and Algorithms , 2022 .

[14]  David J. Crisp,et al.  A Geometric Interpretation of v-SVM Classifiers , 1999, NIPS.

[15]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[16]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[17]  Geoffrey E. Hinton,et al.  Modeling the manifolds of images of handwritten digits , 1997, IEEE Trans. Neural Networks.

[18]  Wei Chu,et al.  An improved conjugate gradient scheme to the solution of least squares SVM , 2005, IEEE Transactions on Neural Networks.

[19]  Shi Yong,et al.  Affine Subspace Nearest Points Classification Algorithm for Wavelet Face Recognition , 2009, 2009 WRI World Congress on Computer Science and Information Engineering.

[20]  Nello Cristianini,et al.  Enlarging the Margins in Perceptron Decision Trees , 2000, Machine Learning.

[21]  Hakan Cevikalp,et al.  Large margin classifiers based on convex class models , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[22]  Johan A. K. Suykens,et al.  Sparse least squares Support Vector Machine classifiers , 2000, ESANN.

[23]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machines , 2002 .

[24]  Christopher J. C. Burges,et al.  A Tutorial on Support Vector Machines for Pattern Recognition , 1998, Data Mining and Knowledge Discovery.

[25]  Nello Cristianini,et al.  Large Margin DAGs for Multiclass Classification , 1999, NIPS.

[26]  A. Martínez,et al.  The AR face databasae , 1998 .

[27]  Hakan Cevikalp,et al.  Nearest hyperdisk methods for high-dimensional classification , 2008, ICML '08.

[28]  David J. Kriegman,et al.  Video-based face recognition using probabilistic appearance manifolds , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[29]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[30]  L. Miles,et al.  2000 , 2000, RDH.

[31]  Chris H. Q. Ding,et al.  R1-PCA: rotational invariant L1-norm principal component analysis for robust subspace factorization , 2006, ICML.

[32]  Chih-Jen Lin,et al.  Training v-Support Vector Classifiers: Theory and Algorithms , 2001, Neural Computation.

[33]  Alexander Gammerman,et al.  Ridge Regression Learning Algorithm in Dual Variables , 1998, ICML.

[34]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.