The Intersection of Two Halfspaces Has High Threshold Degree

The threshold degree of a Boolean function f:{0,1}^n-≫{-1,+1} is the least degree of a real polynomial p such f(x)=sgn p(x). We construct two halfspaces on {0,1}^n whose intersection has threshold degree Theta(sqrt n), an exponential improvement on previous lower bounds. This solves an open problem due to Klivans (2002) and rules out the use of perceptron-based techniques for PAC learning the intersection of two halfspaces, a central unresolved challenge in computational learning. We also prove that the intersection of two majority functions has threshold degree Omega(log n), which is tight and settles a conjecture of O'Donnell and Servedio (2003).

[1]  Alexander A. Sherstov,et al.  Unconditional lower bounds for learning intersections of halfspaces , 2007, Machine Learning.

[2]  Santosh S. Vempala,et al.  A random sampling based algorithm for learning the intersection of half-spaces , 1997, Proceedings 38th Annual Symposium on Foundations of Computer Science.

[3]  Mark Braverman,et al.  Learnability and automatizability , 2004, 45th Annual IEEE Symposium on Foundations of Computer Science.

[4]  Michael E. Saks,et al.  Approximating Threshold Circuits by Rational Functions , 1994, Inf. Comput..

[5]  Scott Aaronson,et al.  BQP and the polynomial hierarchy , 2009, STOC '10.

[6]  Alexander A. Razborov,et al.  Majority gates vs. general weighted threshold gates , 1992, [1992] Proceedings of the Seventh Annual Structure in Complexity Theory Conference.

[7]  Alexander A. Sherstov.  Optimal bounds for sign-representing the intersection of two halfspaces by polynomials , 2009, STOC '10.

[8]  Vladimir V. Podolskii,et al.  Perceptrons of large weight , 2007, Probl. Inf. Transm..

[9]  James Aspnes,et al.  The expressive power of voting polynomials , 1991, STOC '91.

[10]  D. Newman.  Rational approximation to | x , 1964 .

[11]  T. J. Rivlin.  An Introduction to the Approximation of Functions , 2003 .

[12]  Santosh S. Vempala,et al.  An algorithmic theory of learning: Robust concepts and random projection , 1999, Machine Learning.

[13]  Ronald L. Rivest,et al.  Training a 3-node neural network is NP-complete , 1988, COLT '88.

[14]  Alexander A. Sherstov.  The Pattern Matrix Method , 2009, SIAM J. Comput..

[15]  Eric Allender,et al.  A note on the power of threshold circuits , 1989, 30th Annual Symposium on Foundations of Computer Science.

[16]  Ronald de Wolf,et al.  Robust Polynomials and Quantum Algorithms , 2003, Theory of Computing Systems.

[17]  Mark Braverman,et al.  The complexity of properly learning simple concept classes , 2008, J. Comput. Syst. Sci..

[18]  Alexander A. Sherstov.  The pattern matrix method for lower bounds on quantum communication , 2008, STOC '08.

[19]  Vladimir V. Podolskii,et al.  A Uniform Lower Bound on Weights of Perceptrons , 2008, CSR.

[20]  M. Saks.  Slicing the hypercube , 1993 .

[21]  Thomas Kailath,et al.  Rational approximation techniques for analysis of neural networks , 1994, IEEE Trans. Inf. Theory.

[22]  Scott Aaronson.  The Polynomial Method in Quantum and Classical Computing , 2008, 2008 49th Annual IEEE Symposium on Foundations of Computer Science.

[23]  Alexander A. Razborov,et al.  The Sign-Rank of AC0 , 2010, SIAM J. Comput..

[24]  A. Eremenko,et al.  Uniform approximation of sgn x by polynomials and entire functions , 2006, math/0604324.

[25]  Ryan O'Donnell,et al.  Extremal properties of polynomial threshold functions , 2008, J. Comput. Syst. Sci..

[26]  Ronald de Wolf,et al.  Quantum Search on Bounded-Error Inputs , 2003, ICALP.

[27]  Alexander A. Sherstov.  Separating AC0 from depth-2 majority circuits , 2007, STOC '07.

[28]  Yongmei Shi.  Approximating linear restrictions of Boolean functions , 2002 .

[29]  V. Tikhomirov,et al.  DUALITY OF CONVEX FUNCTIONS AND EXTREMUM PROBLEMS , 1968 .

[30]  Adam R. Klivans,et al.  Learning DNF in time 2 Õ(n 1/3 ) . , 2001, STOC 2001.

[31]  Richard Beigel.  Perceptrons, PP, and the polynomial hierarchy , 2005, computational complexity.

[32]  Pavel Pudlák,et al.  On the computational power of depth 2 circuits with threshold and modulo gates , 1994, STOC '94.

[33]  Rocco A. Servedio,et al.  Learning intersections and thresholds of halfspaces , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[34]  Pavel Pudlák,et al.  Computing Boolean functions by polynomials and threshold circuits , 1998, computational complexity.

[35]  Ryan O'Donnell,et al.  Extremal properties of polynomial threshold functions , 2003, 18th IEEE Annual Conference on Computational Complexity, 2003. Proceedings..

[36]  Rocco A. Servedio,et al.  Learning intersections of halfspaces with a margin , 2008, J. Comput. Syst. Sci..

[37]  Stephen Kwek,et al.  PAC Learning Intersections of Halfspaces with Membership Queries , 1998, Algorithmica.

[38]  Rocco A. Servedio,et al.  Learning DNF in time 2Õ(n1/3) , 2004, J. Comput. Syst. Sci..

[39]  Alexander A. Sherstov.  The unbounded-error communication complexity of symmetric functions , 2008, 2008 49th Annual IEEE Symposium on Foundations of Computer Science.

[40]  Alexander A. Sherstov.  Communication Lower Bounds Using Dual Polynomials , 2008, Bull. EATCS.

[41]  Harry Buhrman,et al.  On Computation and Communication with Small Bias , 2007, Twenty-Second Annual IEEE Conference on Computational Complexity (CCC'07).

[42]  Alexander A. Sherstov.  SeparatingAC0 from Depth-2 Majority Circuits , 2009, SIAM J. Comput..

[43]  Subhash Khot,et al.  On hardness of learning intersection of two halfspaces , 2008, STOC '08.

[44]  Nikolai K. Vereshchagin,et al.  Lower bounds for perceptrons solving some separation problems and oracle separation of AM from PP , 1995, Proceedings Third Israel Symposium on the Theory of Computing and Systems.

[45]  P. Gordan.  Ueber die Auflösung linearer Gleichungen mit reellen Coefficienten , 1873 .

[46]  Daniel A. Spielman,et al.  A complexity theoretic approach to learning , 2002 .

[47]  Scott Aaronson,et al.  Quantum computing, postselection, and probabilistic polynomial-time , 2004, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[48]  H. Buhrman,et al.  Complexity measures and decision tree complexity: a survey , 2002, Theor. Comput. Sci..

[49]  Noam Nisan,et al.  On the degree of boolean functions as real polynomials , 1992, STOC '92.

[50]  Andris Ambainis,et al.  Polynomial Degree and Lower Bounds in Quantum Complexity: Collision and Element Distinctness with Small Range , 2003, Theory Comput..

[51]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[52]  Rocco A. Servedio,et al.  Learning intersections of halfspaces with a margin , 2004, J. Comput. Syst. Sci..

[53]  Daniel A. Spielman,et al.  PP is closed under intersection , 1991, STOC '91.

[54]  Troy Lee,et al.  A note on the sign degree of formulas , 2009, ArXiv.

[55]  Alexander A. Razborov,et al.  The Sign-Rank of AC^O , 2008, 2008 49th Annual IEEE Symposium on Foundations of Computer Science.

[56]  Alexander A. Sherstov,et al.  Cryptographic Hardness for Learning Intersections of Halfspaces , 2006, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06).

[57]  Richard Beigel,et al.  The polynomial method in circuit complexity , 1993, [1993] Proceedings of the Eigth Annual Structure in Complexity Theory Conference.

[58]  Alexander A. Sherstov,et al.  The Sign-rank of Ac , 2008 .

[59]  Ryan O'Donnell,et al.  New degree bounds for polynomial threshold functions , 2003, STOC '03.

[60]  Rocco A. Servedio,et al.  Toward Attribute Efficient Learning of Decision Lists and Parities , 2006, J. Mach. Learn. Res..