Unsupervised SVMs: On the Complexity of the Furthest Hyperplane Problem

This paper introduces the Furthest Hyperplane Problem (FHP), which is an unsupervised counterpart of Support Vector Machines. Given a set of n points in R d , the objective is to produce the hyperplane (passing through the origin) which maximizes the separation margin, that is, the minimal distance between the hyperplane and any input point. To the best of our knowledge, this is the first paper achieving provable results regarding FHP. We provide both lower and upper bounds to this NP-hard problem. First, we give a simple randomized algorithm whose running time is n O(1= 2 ) where is the optimal separation margin. We show that its exponential dependency on 1= 2 is tight, up to sub-polynomial factors, assuming SAT cannot be solved in sub-exponential time. Next, we give an efficient approximation algorithm. For any 2 [0; 1], the algorithm produces a hyperplane whose distance from at least 1 3 fraction of the points is at least times the optimal separation margin. Finally, we show that FHP does not admit a PTAS by presenting a gap preserving reduction from a particular version of the PCP theorem.

[1]  Norbert Sauer,et al.  On the Density of Families of Sets , 1972, J. Comb. Theory A.

[2]  O. Mangasarian.  Linear and Nonlinear Separation of Patterns by Linear Programming , 1965 .

[3]  R. Latala.  Some estimates of norms of random matrices , 2005 .

[4]  L. Khachiyan,et al.  The polynomial solvability of convex quadratic programming , 1980 .

[5]  Peter Jonsson,et al.  Hard constraint satisfaction problems have hard gaps at location 1 , 2007, Theor. Comput. Sci..

[6]  V. Vapnik.  Pattern recognition using generalized portrait method , 1963 .

[7]  Santosh S. Vempala,et al.  An algorithmic theory of learning: Robust concepts and random projection , 1999, Machine Learning.

[8]  Ayhan Demiriz,et al.  Semi-Supervised Support Vector Machines , 1998, NIPS.

[9]  Sanjeev Arora.  Probabilistic checking of proofs and hardness of approximation problems , 1995 .

[10]  Noga Alon,et al.  lambda1, Isoperimetric inequalities for graphs, and superconcentrators , 1985, J. Comb. Theory, Ser. B.

[11]  M. Birkner,et al.  Blow-up of semilinear PDE's at the critical dimension. A probabilistic approach , 2002 .

[12]  Rocco A. Servedio,et al.  Learning intersections of halfspaces with a margin , 2004, J. Comput. Syst. Sci..

[13]  G. Lorentz,et al.  Constructive approximation : advanced problems , 1996 .

[14]  Dale Schuurmans,et al.  An efficient algorithm for maximal margin clustering , 2012, J. Glob. Optim..

[15]  Dan Roth,et al.  Maximum Margin Coresets for Active and Noise Tolerant Learning , 2007, IJCAI.

[16]  Santosh S. Vempala,et al.  On Kernels, Margins, and Low-Dimensional Mappings , 2004, ALT.

[17]  M. Murty.  Ramanujan Graphs , 1965 .

[18]  Dale Schuurmans,et al.  Maximum Margin Clustering , 2004, NIPS.

[19]  R. Lata,et al.  SOME ESTIMATES OF NORMS OF RANDOM MATRICES , 2004 .

[20]  Thorsten Joachims,et al.  Transductive Inference for Text Classification using Support Vector Machines , 1999, ICML.

[21]  Nello Cristianini,et al.  Convex Methods for Transduction , 2003, NIPS.