Detecting Patterns Can Be Hard : Circuit Lower Bounds for the Pattern Matching Problem

Detecting patterns in strings and images is a fundamental and widely studied problem. Motivated by the proliferation of specialized circuits in pattern recognition tasks, we study the circuit complexity of pattern matching under two popular choices of gates: De Morgan and threshold gates. For strings of length n and patterns of length log n k ≤ n− o(n), we prove super polynomial lower bounds for De Morgan circuits of depth 2, and nearly linear lower bounds for depth 2 threshold circuits. For unbounded depth and k ≥ 2, we prove a linear lower bound for (unbounded fan-in) De Morgan circuits. For certain values of k, we prove a Ω( √ n/ log n) lower bound for general threshold circuits and a nearly linear lower bound for depth 2 threshold circuits. Our proof for threshold circuits builds on a curious connection between detecting patterns and evaluating Boolean functions when the truth table of the function is given explicitly. Finally, we provide upper bounds on the size of circuits that solve the pattern matching problem. ∗Yahoo Research. Email: alexgolovnev@gmail.com †Department of Electrical Engineering and Computer Science, UC Berkeley. Email: daniel.reichman@gmail.com ‡Department of Electrical Engineering and Computer Science, UC Berkeley. Email: igors@berkeley.edu

[1]  D. Hubel,et al.  Receptive fields and functional architecture of monkey striate cortex , 1968, The Journal of physiology.

[2]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[3]  Saburo Muroga,et al.  Threshold logic and its applications , 1971 .

[4]  Robert S. Boyer,et al.  A fast string searching algorithm , 1977, CACM.

[5]  Donald E. Knuth,et al.  Fast Pattern Matching in Strings , 1977, SIAM J. Comput..

[6]  Zvi Galil.  Optimal Parallel Algorithms for String Matching , 1985, Inf. Control..

[7]  Georg Schnitger,et al.  Parallel Computation with Threshold Functions , 1986, J. Comput. Syst. Sci..

[8]  Pavel Pudlák,et al.  Threshold circuits of bounded depth , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[9]  N. Nisan.  The communication complexity of threshold gates , 1993 .

[10]  Jehoshua Bruck,et al.  Depth efficient neural networks for division and related problems , 1993, IEEE Trans. Inf. Theory.

[11]  D. V. van Essen,et al.  A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[12]  György Turán,et al.  A Liniear lower bound for the size of threshold circuits , 1993, Bull. EATCS.

[13]  Ian Parberry,et al.  Circuit complexity and neural networks , 1994 .

[14]  Alon Orlitsky,et al.  Lower bounds on threshold and related circuits via communication complexity , 1994, IEEE Trans. Inf. Theory.

[15]  Alon Orlitsky,et al.  Neural Models and Spectral Methods , 1994 .

[16]  Terrence J. Sejnowski,et al.  The Computational Brain , 1996, Artif. Intell..

[17]  D. A. Wolfram.  Solving generalized Fibonacci recurrences , 1998 .

[18]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[19]  Robert A. Legenstein,et al.  Foundations for a Circuit Complexity Theory of Sensory Processing , 2000, NIPS.

[20]  Satyanarayana V. Lokam,et al.  Relations Between Communication Complexity, Linear Arrangements, and Computational Complexity , 2001, FSTTCS.

[21]  Gonzalo Navarro,et al.  Flexible Pattern Matching in Strings: Practical On-Line Search Algorithms for Texts and Biological Sequences , 2002 .

[22]  Robert A. Legenstein,et al.  Neural circuits for pattern recognition with small total wire length , 2002, Theor. Comput. Sci..

[23]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[24]  Mark Daniel Ward,et al.  On Correlation Polynomials and Subword Complexity , 2007 .

[25]  Yann LeCun,et al.  Convolutional networks and applications in vision , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[26]  Amit Chakrabarti,et al.  An Optimal Lower Bound on the Communication Complexity of Gap-Hamming-Distance , 2012, SIAM J. Comput..

[27]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[28]  R. Zemel,et al.  On the Representational Efficiency of Restricted Boltzmann Machines , 2013, NIPS 2013.

[29]  Stasys Jukna,et al.  Boolean Function Complexity Advances and Frontiers , 2012, Bull. EATCS.

[30]  Surya Ganguli,et al.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks , 2013, ICLR.

[31]  Xiao Zhou,et al.  Threshold Circuits for Global Patterns in 2-Dimensional Maps , 2015, WALCOM.

[32]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[33]  Ohad Shamir,et al.  The Power of Depth for Feedforward Neural Networks , 2015, COLT.

[34]  Daniel M. Kane,et al.  Super-linear gate and super-quadratic wire lower bounds for depth-two and depth-three threshold circuits , 2015, STOC.

[35]  Surya Ganguli,et al.  On the Expressive Power of Deep Neural Networks , 2016, ICML.