Polynomial Threshold Functions, Hyperplane Arrangements, and Random Tensors

A simple way to generate a Boolean function is to take the sign of a real polynomial in $n$ variables. Such Boolean functions are called polynomial threshold functions. How many low-degree polynomial threshold functions are there? The partial case of this problem for degree $d=1$ was solved by Zuev in 1989, who showed that the number $T(n,1)$ of linear threshold functions satisfies $\log_2 T(n,1) \approx n^2$, up to smaller order terms. However the number of polynomial threshold functions for any higher degrees, including $d=2$, has remained open. We settle this problem for all fixed degrees $d \ge1$, showing that $ \log_2 T(n,d) \approx n \binom{n}{\le d}$. The solution relies on connections between the theory of Boolean threshold functions, hyperplane arrangements, and random tensors. Perhaps surprisingly, it uses also a recent result of E.Abbe, A.Shpilka, and A.Wigderson on Reed-Muller codes.

[1]  Martin Anthony,et al.  Classification by Polynomial Surfaces , 1995, Discret. Appl. Math..

[2]  M. Rudelson,et al.  Hanson-Wright inequality and sub-gaussian concentration , 2013 .

[3]  Alexander A. Sherstov SeparatingAC0 from Depth-2 Majority Circuits , 2009, SIAM J. Comput..

[4]  G. Halász Estimates for the concentration function of combinatorial number theory and probability , 1977 .

[5]  M. Rudelson,et al.  The least singular value of a random square matrix is O(n−1/2) , 2008, 0805.3407.

[6]  Terence Tao,et al.  A sharp inverse Littlewood-Offord theorem , 2010 .

[7]  R. Stanley An Introduction to Hyperplane Arrangements , 2007 .

[8]  Pavel Pudlák,et al.  Computing Boolean functions by polynomials and threshold circuits , 1998, computational complexity.

[9]  Rocco A. Servedio,et al.  Hardness results for agnostically learning low-degree polynomial threshold functions , 2011, SODA '11.

[10]  Rocco A. Servedio,et al.  Learning intersections and thresholds of halfspaces , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[11]  Yaniv Plan,et al.  Dimension Reduction by Random Hyperplane Tessellations , 2014, Discret. Comput. Geom..

[12]  Alexander E. Litvak,et al.  Adjacency matrices of random digraphs: singularity and anti-concentration , 2015, 1511.00113.

[13]  Mark Rudelson,et al.  No-gaps delocalization for general random matrices , 2015, Geometric and Functional Analysis.

[14]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[15]  M. Rudelson,et al.  The Littlewood-Offord problem and invertibility of random matrices , 2007, math/0703503.

[16]  Konstantin E. Tikhomirov The smallest singular value of random rectangular matrices with no moment assumptions on entries , 2014, 1409.7975.

[17]  Ryan O'Donnell,et al.  Analysis of Boolean Functions , 2014, ArXiv.

[18]  Anwar Adkhamovich Irmatov,et al.  On the number of threshold functions , 1993 .

[19]  Gene H. Golub,et al.  Rank-One Approximation to High Order Tensors , 2001, SIAM J. Matrix Anal. Appl..

[20]  Jean Bourgain,et al.  On the singularity probability of discrete random matrices , 2009, 0905.0461.

[21]  G. Rota On the Foundations of Combinatorial Theory , 2009 .

[22]  Richard Beigel,et al.  The polynomial method in circuit complexity , 1993, [1993] Proceedings of the Eigth Annual Structure in Complexity Theory Conference.

[23]  Rocco A. Servedio,et al.  A Regularity Lemma and Low-Weight Approximators for Low-Degree Polynomial Threshold Functions , 2014, Theory Comput..

[24]  Piyush C. Ojha,et al.  Enumeration of linear threshold functions from the lattice of hyperplane intersections , 2000, IEEE Trans. Neural Networks Learn. Syst..

[25]  Hoi H. Nguyen,et al.  On the Singularity of Random Combinatorial Matrices , 2011, SIAM J. Discret. Math..

[26]  Baldi Symmetries and learning in neural network models. , 1987, Physical review letters.

[27]  R. Buck Partition of Space , 1943 .

[28]  Pierre Baldi,et al.  Deep Learning in Biomedical Data Science , 2018, Annual Review of Biomedical Data Science.

[29]  P. Baldi,et al.  Boolean polynomial threshold functions and random tensors , 2018, 1803.10868.

[30]  Kevin P. Costello,et al.  Random symmetric matrices are almost surely nonsingular , 2005, math/0505156.

[31]  Zoltán Füredi,et al.  Solution of the Littlewood-Offord problem in high dimensions , 1988 .

[32]  James Aspnes,et al.  The expressive power of voting polynomials , 1994, Comb..

[33]  Gene H. Golub,et al.  Symmetric Tensors and Symmetric Tensor Rank , 2008, SIAM J. Matrix Anal. Appl..

[34]  Jehoshua Bruck,et al.  Harmonic Analysis of Polynomial Threshold Functions , 1990, SIAM J. Discret. Math..

[35]  Chi Wang,et al.  The threshold order of a Boolean function , 1991, Discret. Appl. Math..

[36]  Rocco A. Servedio,et al.  Learning DNF in time 2Õ(n1/3) , 2004, J. Comput. Syst. Sci..

[37]  Ryan O'Donnell,et al.  New degree bounds for polynomial threshold functions , 2010, Comb..

[38]  Van Vu,et al.  Optimal Inverse Littlewood-Offord theorems , 2010, 1004.3967.

[39]  Konstantin Tikhomirov Sample covariance matrices of heavy-tailed distributions , 2016 .

[40]  M. Anthony Discrete Mathematics of Neural Networks: Selected Topics , 1987 .

[41]  T. Tao,et al.  Random Matrices: the Distribution of the Smallest Singular Values , 2009, 0903.0614.

[42]  Mark Rudelson,et al.  Invertibility of Sparse non-Hermitian matrices , 2015, 1507.03525.

[43]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[44]  Alexander E. Litvak,et al.  Smallest singular value of random matrices with independent columns , 2008 .

[45]  Konstantin Tikhomirov,et al.  The limit of the smallest singular value of random matrices with i.i.d. entries , 2014, 1410.6263.

[46]  J. Littlewood,et al.  On the Number of Real Roots of a Random Algebraic Equation , 1938 .

[47]  B. Bollobás Combinatorics: Set Systems, Hypergraphs, Families of Vectors and Combinatorial Probability , 1986 .

[48]  Nicholas A. Cook On the singularity of adjacency matrices for random regular digraphs , 2014, 1411.0243.

[49]  Hoi H. Nguyen,et al.  On the least singular value of random symmetric matrices , 2011, 1102.1476.

[50]  Noga Alon,et al.  Anti-Hadamard Matrices, Coin Weighing, Threshold Gates, and Indecomposable Hypergraphs , 1997, J. Comb. Theory, Ser. A.

[51]  Johan Håstad,et al.  On the Size of Weights for Threshold Gates , 1994, SIAM J. Discret. Math..

[52]  Pierre Baldi,et al.  Group Actions and Learning for a Family of Automata , 1988, J. Comput. Syst. Sci..

[53]  Pierre Baldi,et al.  The capacity of feedforward neural networks , 2019, Neural Networks.

[54]  T. Zaslavsky Facing Up to Arrangements: Face-Count Formulas for Partitions of Space by Hyperplanes , 1975 .

[55]  L. Schläfli Gesammelte mathematische Abhandlungen , 1950 .

[56]  Daniel A. Spielman,et al.  PP is closed under intersection , 1991, STOC '91.

[57]  E. Szemerédi,et al.  On the probability that a random ±1-matrix is singular , 1995 .

[58]  Jiri Matousek,et al.  Lectures on discrete geometry , 2002, Graduate texts in mathematics.

[59]  P. Erdös On a lemma of Littlewood and Offord , 1945 .

[60]  Friedrich Götze,et al.  Asymptotic expansions for bivariate von Mises functionals , 1979 .

[61]  T. Tao,et al.  Inverse Littlewood-Offord theorems and the condition number of random discrete matrices , 2005, math/0511215.

[62]  R. Winder Partitions of N-Space by Hyperplanes , 1966 .

[63]  Anwar Adkhamovich Irmatov,et al.  Arrangements of Hyperplanes and the Number of Threshold Functions , 2001 .

[64]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry, Expanded Edition , 1987 .

[65]  Timothy M. Chan,et al.  Polynomial Representations of Threshold Functions and Algorithmic Applications , 2016, 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS).

[66]  Saburo Muroga,et al.  Lower Bounds of the Number of Threshold Functions and a Maximum Weight , 1962, IEEE Trans. Electron. Comput..

[67]  Günter M. Ziegler,et al.  Singular 0/1-Matrices, and the Hyperplanes Spanned by Random 0/1-Vectors , 2006, Comb. Probab. Comput..

[68]  T. Tao,et al.  On the singularity probability of random Bernoulli matrices , 2005, math/0501313.

[69]  E. Giné,et al.  Decoupling: From Dependence to Independence , 1998 .

[70]  Roman Vershynin,et al.  High-Dimensional Probability , 2018 .

[71]  M. Talagrand A new look at independence , 1996 .

[72]  J. G. Wendel A Problem in Geometric Probability. , 1962 .

[73]  T. Tao,et al.  On random ±1 matrices: Singularity and determinant , 2006 .

[74]  Pierre Baldi,et al.  On neuronal capacity , 2019, NeurIPS.

[75]  M. Rudelson Invertibility of random matrices: norm of the inverse , 2005, math/0507024.

[76]  Thomas M. Cover,et al.  Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition , 1965, IEEE Trans. Electron. Comput..

[77]  Nigel J. Kalton,et al.  Rademacher series and decoupling , 2005 .

[78]  Roman Vershynin,et al.  Invertibility of symmetric random matrices , 2011, Random Struct. Algorithms.

[79]  M. Rudelson,et al.  Non-asymptotic theory of random matrices: extreme singular values , 2010, 1003.2990.

[80]  Jovisa D. Zunic On encoding and enumerating threshold functions , 2004, IEEE Transactions on Neural Networks.

[81]  Mark Rudelson,et al.  Invertibility of random matrices: Unitary and orthogonal perturbations , 2012, 1206.5180.

[82]  M. Ledoux The concentration of measure phenomenon , 2001 .

[83]  Andrew M. Odlyzko,et al.  On subspaces spanned by random selections of plus/minus 1 vectors , 1988, Journal of combinatorial theory. Series A.

[84]  I. Bárány LECTURES ON DISCRETE GEOMETRY (Graduate Texts in Mathematics 212) , 2003 .

[85]  T. Tao,et al.  From the Littlewood-Offord problem to the Circular Law: universality of the spectral distribution of random matrices , 2008, 0810.2994.

[86]  P. McCullagh Tensor Methods in Statistics , 1987 .

[87]  Raghu Meka,et al.  Anti-concentration for Polynomials of Independent Random Variables , 2016, Theory Comput..

[88]  Hoi H. Nguyen,et al.  Inverse Littlewood-Offord problems and The Singularity of Random Symmetric Matrices , 2011, 1101.3074.

[89]  Ryan O'Donnell,et al.  Extremal properties of polynomial threshold functions , 2008, J. Comput. Syst. Sci..

[90]  M. Saks Slicing the hypercube , 1993 .

[91]  Alexey Naumov,et al.  On minimal singular values of random matrices with correlated entries , 2013, 1309.5711.

[92]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[93]  M. Rudelson,et al.  The smallest singular value of a random rectangular matrix , 2008, 0802.3956.

[94]  Pierre Baldi,et al.  Neural networks, orientations of the hypercube, and algebraic threshold functions , 1988, IEEE Trans. Inf. Theory.

[95]  D. Kane A structure theorem for poorly anticoncentrated polynomials of Gaussians and applications to the study of polynomial threshold functions , 2017 .