Super-linear gate and super-quadratic wire lower bounds for depth-two and depth-three threshold circuits

In order to formally understand the power of neural computing, we first need to crack the frontier of threshold circuits with two and three layers, a regime that has been surprisingly intractable to analyze. We prove the first super-linear gate lower bounds and the first super-quadratic wire lower bounds for depth-two linear threshold circuits with arbitrary weights, and depth-three majority circuits computing an explicit function. (1) We prove that for all ε ≪ √log(n)/n, the linear-time computable Andreev’s function cannot be computed on a (1/2+ε)-fraction of n-bit inputs by depth-two circuits of o(ε3 n3/2/log3 n) gates, nor can it be computed with o(ε3 n5/2/log7/2 n) wires. This establishes an average-case “size hierarchy” for threshold circuits, as Andreev’s function is computable by uniform depth-two circuits of o(n3) linear threshold gates, and by uniform depth-three circuits of O(n) majority gates. (2) We present a new function in P based on small-biased sets, which we prove cannot be computed by a majority vote of depth-two threshold circuits of o(n3/2/log3 n) gates, nor with o(n5/2/log7/2n) wires. (3) We give tight average-case (gate and wire) complexity results for computing PARITY with depth-two threshold circuits; the answer turns out to be the same as for depth-two majority circuits. The key is a new method for analyzing random restrictions to linear threshold functions. Our main analytical tool is the Littlewood-Offord Lemma from additive combinatorics.

[1]  J. Littlewood,et al.  On the Number of Real Roots of a Random Algebraic Equation , 1938 .

[2]  J. Littlewood,et al.  On the number of real roots of a random algebraic equation. II , 1939 .

[3]  P. Erdös.  On a lemma of Littlewood and Offord , 1945 .

[4]  C. K. Chow,et al.  On the characterization of threshold functions , 1961, SWCT.

[5]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[6]  Robert O. Winder,et al.  Threshold logic , 1971, IEEE Spectrum.

[7]  References , 1971 .

[8]  Saburo Muroga,et al.  Threshold logic and its applications , 1971 .

[9]  Andrew Chi-Chih Yao,et al.  Separating the Polynomial-Time Hierarchy by Oracles (Preliminary Version) , 1985, FOCS.

[10]  Avi Wigderson,et al.  Deterministic simulation of probabilistic constant depth circuits , 1985, 26th Annual Symposium on Foundations of Computer Science (sfcs 1985).

[11]  Avi Wigderson,et al.  Deterministic Simulation of Probabilistic Constant Depth Circuits (Preliminary Version) , 1985, FOCS 1985.

[12]  G. Palm.  Warren McCulloch and Walter Pitts: A Logical Calculus of the Ideas Immanent in Nervous Activity , 1986 .

[13]  Johan Håstad,et al.  Almost optimal lower bounds for small depth circuits , 1986, STOC '86.

[14]  Pavel Pudlák,et al.  Threshold circuits of bounded depth , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[15]  Moni Naor,et al.  Decision trees and downward closures , 1988, [1988] Proceedings. Structure in Complexity Theory Third Annual Conference.

[16]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[17]  Noga Alon,et al.  Simple construction of almost k-wise independent random variables , 1990, Proceedings [1990] 31st Annual Symposium on Foundations of Computer Science.

[18]  György Turán,et al.  On Linear Decision Trees Computing Boolean Functions , 1991, ICALP.

[19]  Uri Zwick,et al.  Shrinkage of de Morgan formulae under restriction , 1991, [1991] Proceedings 32nd Annual Symposium of Foundations of Computer Science.

[20]  Noga Alon,et al.  Simple Construction of Almost k-wise Independent Random Variables , 1992, Random Struct. Algorithms.

[21]  Alexander A. Razborov,et al.  Majority gates vs. general weighted threshold gates , 1992, [1992] Proceedings of the Seventh Annual Structure in Complexity Theory Conference.

[22]  N. Nisan.  The communication complexity of threshold gates , 1993 .

[23]  György Turán,et al.  A Liniear lower bound for the size of threshold circuits , 1993, Bull. EATCS.

[24]  Michael E. Saks,et al.  Approximating Threshold Circuits by Rational Functions , 1994, Inf. Comput..

[25]  K. Siu,et al.  Theoretical Advances in Neural Computation and Learning , 1994, Springer US.

[26]  Alon Orlitsky,et al.  Lower bounds on threshold and related circuits via communication complexity , 1994, IEEE Trans. Inf. Theory.

[27]  Alon Orlitsky,et al.  Neural Models and Spectral Methods , 1994 .

[28]  Michael E. Saks,et al.  Size-depth trade-offs for threshold circuits , 1993, SIAM J. Comput..

[29]  Johan Hå stad.  The Shrinkage Exponent of de Morgan Formulas is 2 , 1998 .

[30]  J. Håstad.  The Shrinkage Exponent of de Morgan Formulas is 2 , 1998, SIAM J. Comput..

[31]  Satyanarayana V. Lokam,et al.  Relations Between Communication Complexity, Linear Arrangements, and Computational Complexity , 2001, FSTTCS.

[32]  David Thomas,et al.  The Art in Computer Programming , 2001 .

[33]  Akira Maruoka,et al.  On the Complexity of Depth-2 Circuits with Threshold Gates , 2005, MFCS.

[34]  Michael Sipser,et al.  Parity, circuits, and the polynomial-time hierarchy , 1981, 22nd Annual Symposium on Foundations of Computer Science (sfcs 1981).

[35]  Eric Allender,et al.  Amplifying Lower Bounds by Means of Self-Reducibility , 2008, 2008 23rd Annual IEEE Conference on Computational Complexity.

[36]  Benjamin Rossman,et al.  On the constant-depth complexity of k-clique , 2008, STOC.

[37]  Ryan O'Donnell,et al.  The chow parameters problem , 2008, SIAM J. Comput..

[38]  Kristoffer Arnsfelt Hansen,et al.  Exact Threshold Circuits , 2010, 2010 IEEE 25th Annual Conference on Computational Complexity.

[39]  Rahul Santhanam,et al.  Fighting Perebor: New and Improved Algorithms for Formula and QBF Satisfiability , 2010, 2010 IEEE 51st Annual Symposium on Foundations of Computer Science.

[40]  Donald E. Knuth,et al.  The Art of Computer Programming: Combinatorial Algorithms, Part 1 , 2011 .

[41]  Johan Håstad,et al.  On the Correlation of Parity and Small-Depth Circuits , 2014, SIAM J. Comput..

[42]  Kazuhisa Seto,et al.  A satisfiability algorithm and average-case hardness for formulas over the full binary basis , 2012, computational complexity.

[43]  Russell Impagliazzo,et al.  Pseudorandomness from Shrinkage , 2012, 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science.

[44]  Ran Raz,et al.  Improved Average-Case Lower Bounds for DeMorgan Formula Size , 2013, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[45]  Russell Impagliazzo,et al.  A Satisfiability Algorithm for Sparse Depth Two Threshold Circuits , 2012, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[46]  Kristoffer Arnsfelt Hansen,et al.  Polynomial threshold functions and Boolean threshold circuits , 2013, Inf. Comput..

[47]  Ran Raz,et al.  Average-case lower bounds for formula size , 2013, STOC '13.

[48]  David Zuckerman,et al.  Mining Circuit Lower Bound Proofs for Meta-Algorithms , 2014, computational complexity.

[49]  Ryan Williams,et al.  New algorithms and lower bounds for circuits with linear threshold gates , 2014, STOC.

[50]  Rocco A. Servedio,et al.  An Average-Case Depth Hierarchy Theorem for Boolean Circuits , 2015, 2015 IEEE 56th Annual Symposium on Foundations of Computer Science.

[51]  Rahul Santhanam,et al.  Average-Case Lower Bounds and Satisfiability Algorithms for Small Threshold Circuits , 2016, CCC.

[52]  Huacheng Yu,et al.  More Applications of the Polynomial Method to Algorithm Design , 2015, SODA.

[53]  Valentine Kabanets,et al.  An Improved Deterministic #SAT Algorithm for Small de Morgan Formulas , 2014, Algorithmica.

[54]  Rahul Santhanam,et al.  Improved Algorithms for Sparse MAX-SAT and MAX-k-CSP , 2015, SAT.

[55]  Richard Ryan Williams.  New Algorithms and Lower Bounds for Circuits With Linear Threshold Gates , 2018, Theory Comput..