Coresets for polytope distance

Following recent work of Clarkson, we translate the coreset framework to the problems of finding the point closest to the origin inside a polytope, finding the shortest distance between two polytopes, Perceptrons, and soft- as well as hard-margin Support Vector Machines (SVM). We prove asymptotically matching upper and lower bounds on the size of coresets, stating that µ-coresets of size (1+o(1)) E*/µ do always exist as µ-0, and that this is best possible. The crucial quantity E* is what we call the excentricity of a polytope, or a pair of polytopes. Additionally, we prove linear convergence speed of Gilbert's algorithm, one of the earliest known approximation algorithms for polytope distance, and generalize both the algorithm and the proof to the two polytope case. Interestingly, our coreset bounds also imply that we can for the first time prove matching upper and lower bounds for the sparsity of Perceptron and SVM solutions.

[1]  Philip Wolfe,et al.  An algorithm for quadratic programming , 1956 .

[2]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[3]  Albert B Novikoff,et al.  ON CONVERGENCE PROOFS FOR PERCEPTRONS , 1963 .

[4]  E. Gilbert.  An Iterative Procedure for Computing the Minimum of a Quadratic Form on a Convex Set , 1966 .

[5]  V. N. Malozemov,et al.  Finding the Point of a Polyhedron Closest to the Origin , 1974 .

[6]  S. Sathiya Keerthi,et al.  A fast procedure for computing the distance between complex objects in three-dimensional space , 1988, IEEE J. Robotics Autom..

[7]  Bart Kosko,et al.  Neural networks for signal processing , 1992 .

[8]  G. Ziegler.  Lectures on Polytopes , 1994 .

[9]  J. C. BurgesChristopher.  A Tutorial on Support Vector Machines for Pattern Recognition , 1998 .

[10]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[11]  David J. Crisp,et al.  A Geometric Interpretation of ?-SVM Classifiers , 1999, NIPS 2000.

[12]  S. Sathiya Keerthi,et al.  A fast iterative nearest point algorithm for support vector machine classifier design , 2000, IEEE Trans. Neural Networks Learn. Syst..

[13]  Thore Graepel,et al.  From Margin to Sparsity , 2000, NIPS.

[14]  Kristin P. Bennett,et al.  Duality and Geometry in SVM Classifiers , 2000, ICML.

[15]  D. Roobaert.  DirectSVM: a fast and simple support vector machine perceptron , 2000, Neural Networks for Signal Processing X. Proceedings of the 2000 IEEE Signal Processing Society Workshop (Cat. No.00TH8501).

[16]  Piotr Indyk,et al.  Approximate clustering via core-sets , 2002, STOC '02.

[17]  Kenneth L. Clarkson,et al.  Smaller core-sets for balls , 2003, SODA '03.

[18]  Pankaj K. Agarwal,et al.  Approximating extent measures of points , 2004, JACM.

[19]  Rina Panigrahy,et al.  Minimum Enclosing Polytope in High Dimensions , 2004, ArXiv.

[20]  Ivor W. Tsang,et al.  Core Vector Machines: Fast SVM Training on Very Large Data Sets , 2005, J. Mach. Learn. Res..

[21]  Yoram Singer,et al.  A New Perspective on an Old Perceptron Algorithm , 2005, COLT.

[22]  Sergios Theodoridis,et al.  A geometric approach to Support Vector Machine (SVM) classification , 2006, IEEE Transactions on Neural Networks.

[23]  Jacek M. Zurada,et al.  Generalized Core Vector Machines , 2006, IEEE Transactions on Neural Networks.

[24]  Sergios Theodoridis,et al.  A novel SVM Geometric Algorithm based on Reduced Convex Hulls , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[25]  I. Tsang,et al.  Simpler core vector machines with enclosing balls , 2007, ICML '07.

[26]  Kasturi R. Varadarajan,et al.  Geometric Approximation via Coresets , 2007 .

[27]  Dan Roth,et al.  Maximum Margin Coresets for Active and Noise Tolerant Learning , 2007, IJCAI.

[28]  Michael J. Todd,et al.  On Khachiyan's algorithm for the computation of minimum-volume enclosing ellipsoids , 2007, Discret. Appl. Math..

[29]  Kenneth L. Clarkson,et al.  Coresets, sparse greedy approximation, and the Frank-Wolfe algorithm , 2008, SODA '08.

[30]  José R. Dorronsoro,et al.  On the Equivalence of the SMO and MDM Algorithms for SVM Training , 2008, ECML/PKDD.

[31]  Peng Sun,et al.  Linear convergence of a modified Frank–Wolfe algorithm for computing minimum-volume enclosing ellipsoids , 2008, Optim. Methods Softw..

[32]  Kenneth L. Clarkson,et al.  Optimal core-sets for balls , 2008, Comput. Geom..