Multiconlitron: A General Piecewise Linear Classifier

Based on the “convexly separable” concept, we present a solid geometric theory and a new general framework to design piecewise linear classifiers for two arbitrarily complicated nonintersecting classes by using a “multiconlitron,” which is a union of multiple conlitrons that comprise a set of hyperplanes or linear functions surrounding a convex region for separating two convexly separable datasets. We propose a new iterative algorithm called the cross distance minimization algorithm (CDMA) to compute hard margin non-kernel support vector machines (SVMs) via the nearest point pair between two convex polytopes. Using CDMA, we derive two new algorithms, i.e., the support conlitron algorithm (SCA) and the support multiconlitron algorithm (SMA) to construct support conlitrons and support multiconlitrons, respectively, which are unique and can separate two classes by a maximum margin as in an SVM. Comparative experiments show that SMA can outperform linear SVM on many of the selected databases and provide similar results to radial basis function SVM on some of them, while SCA performs better than linear SVM on three out of four applicable databases. Other experiments show that SMA and SCA may be further improved to draw more potential in the new research direction of piecewise linear learning.

[1]  Xinhua Zhuang,et al.  Piecewise linear classifiers using binary tree structure and genetic algorithm , 1996, Pattern Recognit..

[2]  I. Song,et al.  Working Set Selection Using Second Order Information for Training Svm, " Complexity-reduced Scheme for Feature Extraction with Linear Discriminant Analysis , 2022 .

[3]  James A. Bucklew,et al.  Support vector machine techniques for nonlinear equalization , 2000, IEEE Trans. Signal Process..

[4]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[5]  Peter Williams,et al.  A Geometrical Method to Improve Performance of the Support Vector Machine , 2007, IEEE Transactions on Neural Networks.

[6]  Gunnar Rätsch,et al.  A General and Efficient Multiple Kernel Learning Algorithm , 2005, NIPS.

[7]  S. Sathiya Keerthi,et al.  Computation of certain measures of proximity between convex polytopes: a complexity viewpoint , 1992, Proceedings 1992 IEEE International Conference on Robotics and Automation.

[8]  O. Mangasarian,et al.  Pattern Recognition Via Linear Programming: Theory and Application to Medical Diagnosis , 1989 .

[9]  David R. Musicant,et al.  Lagrangian Support Vector Machines , 2001, J. Mach. Learn. Res..

[10]  Olvi L. Mangasarian,et al.  Multisurface proximal support vector machine classification via generalized eigenvalues , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Kristin P. Bennett,et al.  Duality and Geometry in SVM Classifiers , 2000, ICML.

[12]  Alexander E. Kostin,et al.  A simple and fast multi-class piecewise linear pattern classifier , 2006, Pattern Recognit..

[13]  David Masip,et al.  Geometry-Based Ensembles: Toward a Structural Characterization of the Classification Boundary , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[15]  D Haussler,et al.  Knowledge-based analysis of microarray gene expression data by using support vector machines. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[16]  Reshma Khemchandani,et al.  Twin Support Vector Machines for Pattern Classification , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Patrick Gallinari,et al.  SGD-QN: Careful Quasi-Newton Stochastic Gradient Descent , 2009, J. Mach. Learn. Res..

[18]  S. Sathiya Keerthi,et al.  A fast iterative nearest point algorithm for support vector machine classifier design , 2000, IEEE Trans. Neural Networks Learn. Syst..

[19]  S. Sathiya Keerthi,et al.  Improvements to Platt's SMO Algorithm for SVM Classifier Design , 2001, Neural Computation.

[20]  Songcan Chen,et al.  MultiK-MHKS: A Novel Multiple Kernel Learning Algorithm , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Chih-Jen Lin,et al.  A study on reduced support vector machines , 2003, IEEE Trans. Neural Networks.

[22]  Constantin Zopounidis,et al.  Additive Support Vector Machines for Pattern Classification , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[23]  Sergios Theodoridis,et al.  A geometric approach to Support Vector Machine (SVM) classification , 2006, IEEE Transactions on Neural Networks.

[24]  Huanhuan Chen,et al.  Probabilistic Classification Vector Machines , 2009, IEEE Transactions on Neural Networks.

[25]  Yoram Singer,et al.  Pegasos: primal estimated sub-gradient solver for SVM , 2011, Math. Program..

[26]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[27]  Atsuyoshi Nakamura,et al.  Classification by reflective convex hulls , 2008, 2008 19th International Conference on Pattern Recognition.

[28]  Václav Hlavác,et al.  An iterative algorithm learning the maximal margin classifier , 2003, Pattern Recognit..

[29]  Olvi L. Mangasarian,et al.  Polyhedral Boundary Projection , 1999, SIAM J. Optim..

[30]  Glenn Fung,et al.  Proximal support vector machine classifiers , 2001, KDD '01.

[31]  Nikolas P. Galatsanos,et al.  A support vector machine approach for detection of microcalcifications , 2002, IEEE Transactions on Medical Imaging.

[32]  Thorsten Joachims,et al.  Training linear SVMs in linear time , 2006, KDD '06.

[33]  George Eastman House,et al.  Sparse Bayesian Learning and the Relevan e Ve tor Ma hine , 2001 .

[34]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[35]  Fernando Pérez-Cruz,et al.  Weighted least squares training of support vector classifiers leading to compact and adaptive schemes , 2001, IEEE Trans. Neural Networks.

[36]  Sergios Theodoridis,et al.  A Geometric Nearest Point Algorithm for the Efficient Solution of the SVM Classification Task , 2007, IEEE Transactions on Neural Networks.

[37]  Federico Girosi,et al.  Training support vector machines: an application to face detection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[38]  Gunnar Rätsch,et al.  Large Scale Multiple Kernel Learning , 2006, J. Mach. Learn. Res..

[39]  Gabor T. Herman,et al.  On Piecewise-Linear Classification , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Olivier Chapelle,et al.  Training a Support Vector Machine in the Primal , 2007, Neural Computation.

[41]  J. Sklansky,et al.  Automated design of multiple-class piecewise linear classifiers , 1989 .

[42]  Gunnar Rätsch,et al.  An introduction to kernel-based learning algorithms , 2001, IEEE Trans. Neural Networks.

[43]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[44]  Mineichi Kudo,et al.  Piecewise linear classifiers with an appropriate number of hyperplanes , 1998, Pattern Recognit..

[45]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2004 .

[46]  Jack Sklansky,et al.  Locally Trained Piecewise Linear Classifiers , 1980, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Jian-xiong Dong,et al.  Fast SVM training algorithm with decomposition on very large data sets , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Yi Li,et al.  The Relaxed Online Maximum Margin Algorithm , 1999, Machine Learning.