Estimating the Optimal Margins of Embeddings in Euclidean Half Spaces

AbstractConcept classes can canonically be represented by matrices with entries 1 and −1. We use the singular value decomposition of this matrix to determine the optimal margins of embeddings of the concept classes of singletons and of half intervals in homogeneous Euclidean half spaces. For these concept classes the singular value decomposition can be used to construct optimal embeddings and also to prove the corresponding best possible upper bounds on the margin. We show that the optimal margin for embedding n singletons is $$\tfrac{n}{{3n - 4}}$$ and that the optimal margin for half intervals over {1,...,n} is $$\tfrac{\pi }{{2\ln n}} + \Theta (\tfrac{1}{{(\ln n)^2 }})$$ . For the upper bounds on the margins we generalize a bound by Forster (2001). We also determine the optimal margin of some concept classes defined by circulant matrices up to a small constant factor, and we discuss the concept classes of monomials to point out limitations of our approach.

[1]  Matthias Krause,et al.  Geometric arguments yield better bounds for threshold circuits and distributed computing , 1991, [1991] Proceedings of the Sixth Annual Structure in Complexity Theory Conference.

[2]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[3]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[4]  Albert B Novikoff,et al.  ON CONVERGENCE PROOFS FOR PERCEPTRONS , 1963 .

[5]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[6]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[7]  Wolfgang Maass,et al.  Lower Bound Methods and Separation Results for On-Line Learning Models , 2004, Machine Learning.

[8]  Satyanarayana V. Lokam,et al.  Relations Between Communication Complexity, Linear Arrangements, and Computational Complexity , 2001, FSTTCS.

[9]  Nello Cristianini,et al.  An introduction to Support Vector Machines , 2000 .

[10]  Jürgen Forster.  A linear lower bound on the unbounded error probabilistic communication complexity , 2002, J. Comput. Syst. Sci..

[11]  Umesh V. Vazirani,et al.  An Introduction to Computational Learning Theory , 1994 .

[12]  Shai Ben-David,et al.  Limitations of Learning Via Embeddings in Euclidean Half Spaces , 2003, J. Mach. Learn. Res..

[13]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.