Sign rank, VC dimension and spectral gaps

We study the maximum possible sign rank of N×N sign matrices with a given VC dimension d. For d = 1, this maximum is 3. For d = 2, this maximum is Θ(N1/2). Similar (slightly less accurate) statements hold for d > 2 as well. We discuss the tightness of our methods, and describe connections to combinatorics, communication complexity and learning theory. We also provide explicit examples of matrices with low VC dimension and high sign rank. Let A be the N × N point-hyperplane incidence matrix of a finite projective geometry with order n ≥ 3 and dimension d ≥ 2. The VC dimension of A is d, and we prove that its sign rank is larger than N 1 2 − 1 2d . The large sign rank of A demonstrates yet another difference between finite and real geometries. To analyse the sign rank of A, we introduce a connection between sign rank and spectral gaps, which may be of independent interest. Consider the N × N adjacency matrix of a ∆ regular graph with a second eigenvalue in absolute value λ and ∆ ≤ N/2. We show that the sign rank of the signed version of this matrix is at least ∆/λ. A similar statement holds for all regular (not necessarily symmetric) sign matrices. We also describe limitations of this approach, in the spirit of the Alon-Boppana theorem.

[1]  W. G. Brown.  On Graphs that do not Contain a Thomsen Graph , 1966, Canadian Mathematical Bulletin.

[2]  H. Warren.  Lower bounds for approximation by nonlinear manifolds , 1968 .

[3]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[4]  Norbert Sauer,et al.  On the Density of Families of Sets , 1972, J. Comb. Theory A.

[5]  J. Dodziuk.  Difference equations, isoperimetric inequality and transience of certain random walks , 1984 .

[6]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[7]  W. B. Johnson,et al.  Extensions of Lipschitz mappings into Hilbert space , 1984 .

[8]  Vojtech Rödl,et al.  Geometrical realization of set systems and probabilistic communication complexity , 1985, 26th Annual Symposium on Foundations of Computer Science (sfcs 1985).

[9]  N. Alon,et al.  il , , lsoperimetric Inequalities for Graphs , and Superconcentrators , 1985 .

[10]  N. Alon.  Eigenvalues and expanders , 1986, Comb..

[11]  David Haussler,et al.  Classifying learnable geometric concepts with the Vapnik-Chervonenkis dimension , 1986, STOC '86.

[12]  Noga Alon,et al.  Eigenvalues, geometric expanders, sorting in rounds, and ramsey theory , 1986, Comb..

[13]  Fan Chung Graham,et al.  Some intersection theorems for ordered sets and graphs , 1986, J. Comb. Theory, Ser. A.

[14]  Janos Simon,et al.  Probabilistic Communication Complexity , 1986, J. Comput. Syst. Sci..

[15]  David Haussler,et al.  Epsilon-nets and simplex range queries , 1986, SCG '86.

[16]  Noga Alon,et al.  Partitioning and geometric embedding of range spaces of finite Vapnik-Chervonenkis dimension , 1987, SCG '87.

[17]  Emo Welzl,et al.  Partition trees for triangle counting and other range searching problems , 1988, SCG '88.

[18]  Bernard Chazelle,et al.  Quasi-optimal range searching in spaces of finite VC-dimension , 1989, Discret. Comput. Geom..

[19]  Noga Alon,et al.  On the second eigenvalue of a graph , 1991, Discret. Math..

[20]  J. Komlos,et al.  Almost tight bounds for $\epsilon$-nets , 1992 .

[21]  A. Beutelspacher,et al.  Projective Geometrie : Von den Grundlagen bis zu den Anwendungen , 1992 .

[22]  Jirí Matousek,et al.  Discrepancy and approximations for bounded VC-dimension , 1993, Comb..

[23]  David Haussler,et al.  Sphere Packing Numbers for Subsets of the Boolean n-Cube with Bounded Vapnik-Chervonenkis Dimension , 1995, J. Comb. Theory, Ser. A.

[24]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[25]  J. C. BurgesChristopher.  A Tutorial on Support Vector Machines for Pattern Recognition , 1998 .

[26]  Noam Nisan,et al.  On Randomized One-round Communication Complexity , 1995, STOC '95.

[27]  Noga Alon,et al.  Norm-Graphs: Variations and Applications , 1999, J. Comb. Theory, Ser. B.

[28]  Shai Ben-David,et al.  Limitations of Learning Via Embeddings in Euclidean Half Spaces , 2003, J. Mach. Learn. Res..

[29]  Adam R. Klivans,et al.  Learning DNF in time 2 Õ(n 1/3 ) . , 2001, STOC 2001.

[30]  Jürgen Forster,et al.  A linear lower bound on the unbounded error probabilistic communication complexity , 2001, Proceedings 16th Annual IEEE Conference on Computational Complexity.

[31]  Satyanarayana V. Lokam,et al.  Relations Between Communication Complexity, Linear Arrangements, and Computational Complexity , 2001, FSTTCS.

[32]  Hans Ulrich Simon,et al.  Estimating the Optimal Margins of Embeddings in Euclidean Half Spaces , 2004, Machine Learning.

[33]  Rocco A. Servedio,et al.  Learning DNF in time 2Õ(n1/3) , 2004, J. Comput. Syst. Sci..

[34]  N. Linial,et al.  Expander Graphs and their Applications , 2006 .

[35]  Hans Ulrich Simon,et al.  On the smallest possible dimension and the largest possible margin of linear arrangements representing given concept classes , 2006, Theor. Comput. Sci..

[36]  Alexander A. Sherstov.  Halfspace Matrices , 2007, Twenty-Second Annual IEEE Conference on Computational Complexity (CCC'07).

[37]  Alexander A. Sherstov.  Communication Complexity Under Product and Nonproduct Distributions , 2008, 2008 23rd Annual IEEE Conference on Computational Complexity.

[38]  Alexander A. Razborov,et al.  The Sign-Rank of AC^O , 2008, 2008 49th Annual IEEE Symposium on Foundations of Computer Science.

[39]  Nathan Linial,et al.  Learning Complexity vs. Communication Complexity , 2008, 2008 23rd Annual IEEE Conference on Computational Complexity.

[40]  Hans Ulrich Simon,et al.  Recursive Teaching Dimension, Learning Complexity, and Maximum Classes , 2010, ALT.

[41]  A. Razborov.  Communication Complexity , 2011 .

[42]  Noga Alon,et al.  The structure of almost all graphs in a hereditary property , 2009, J. Comb. Theory B.

[43]  Heinz Luneburg.  Projektive Geometrie , 2011, 1106.5691.