On hardness of learning intersection of two halfspaces

We show that unless NP = RP, it is hard to (even) weakly PAC-learn intersection of two halfspaces in Rn using a hypothesis which is a function of up to l linear threshold functions for any integer l. Specifically, we show that for every integer l and an arbitrarily small constant ε > 0, unless NP = RP, no polynomial time algorithm can distinguish whether there is an intersection of two halfspaces that correctly classifies a given set of labeled points in Rn, or whether any function of l linear threshold functions can correctly classify at most 1/2+ε fraction of the points.

[1]  Ronald L. Rivest,et al.  Training a 3-node neural network is NP-complete , 1988, COLT '88.

[2]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[3]  Robert E. Schapire,et al.  The strength of weak learnability , 1990, Mach. Learn..

[4]  I. Benjamini,et al.  Noise sensitivity of Boolean functions and applications to percolation , 1998 .

[5]  Vitaly Feldman,et al.  New Results for Learning Noisy Parities and Halfspaces , 2006, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06).

[6]  Sanjeev Arora,et al.  Probabilistic checking of proofs; a new characterization of NP , 1992, Proceedings., 33rd Annual Symposium on Foundations of Computer Science.

[7]  Santosh S. Vempala,et al.  A random-sampling-based algorithm for learning intersections of halfspaces , 2010, JACM.

[8]  Subhash Khot,et al.  A new PCP outer verifier with applications to homogeneous linear equations and max-bisection , 2004, STOC '04.

[9]  O. J. Murphy,et al.  Nearest neighbor pattern classification perceptrons , 1990, Proc. IEEE.

[10]  Alan M. Frieze,et al.  A Polynomial-Time Algorithm for Learning Noisy Linear Threshold Functions , 1996, Algorithmica.

[11]  Rocco A. Servedio,et al.  Learning intersections and thresholds of halfspaces , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[12]  Santosh S. Vempala,et al.  A random sampling based algorithm for learning the intersection of half-spaces , 1997, Proceedings 38th Annual Symposium on Foundations of Computer Science.

[13]  I. Benjamini,et al.  Noise sensitivity of Boolean functions and applications to percolation , 1998, math/9811157.

[14]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.

[15]  Rocco A. Servedio,et al.  Learning intersections of halfspaces with a margin , 2004, J. Comput. Syst. Sci..

[16]  Subhash Khot,et al.  A 3-query non-adaptive PCP with perfect completeness , 2006, 21st Annual IEEE Conference on Computational Complexity (CCC'06).

[17]  Owen Murphy,et al.  Computing Nearest Neighbor Pattern Classification Perceptrons , 1995, Inf. Sci..

[18]  Subhash Khot,et al.  Hardness of Reconstructing Multivariate Polynomials over Finite Fields , 2007, FOCS.

[19]  Prasad Raghavendra,et al.  Hardness of Learning Halfspaces with Noise , 2006, FOCS.

[20]  Alexander A. Sherstov,et al.  Cryptographic Hardness for Learning Intersections of Halfspaces , 2006, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06).

[21]  Carsten Lund,et al.  Proof verification and the hardness of approximation problems , 1998, JACM.

[22]  Amnon Shashua,et al.  Principal Component Analysis over Continuous Subspaces and Intersection of Half-Spaces , 2002, ECCV.

[23]  Mark Braverman,et al.  Learnability and automatizability , 2004, 45th Annual IEEE Symposium on Foundations of Computer Science.

[24]  Santosh S. Vempala,et al.  An algorithmic theory of learning: Robust concepts and random projection , 1999, Machine Learning.

[25]  Ran Raz,et al.  A parallel repetition theorem , 1995, STOC '95.

[26]  Avrim Blum,et al.  Learning an Intersection of a Constant Number of Halfspaces over a Uniform Distribution , 1997, J. Comput. Syst. Sci..

[27]  Rocco A. Servedio,et al.  Agnostically learning halfspaces , 2005, 46th Annual IEEE Symposium on Foundations of Computer Science (FOCS'05).

[28]  Subhash Khot,et al.  Hardness results for coloring 3-colorable 3-uniform hypergraphs , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[29]  Stefan Kramer,et al.  Quantitative association rules based on half-spaces: an optimization approach , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).