Superlinear Lower Bounds for Multipass Graph Processing

We prove $$n^{1+\varOmega (1/p)}/p^{O(1)}$$n1+Ω(1/p)/pO(1) lower bounds for the space complexity of p-pass streaming algorithms solving the following problems on n-vertex graphs: (1) testing if an undirected graph has a perfect matching (this implies lower bounds for computing a maximum matching or even just the maximum matching size), (2) testing if two specific vertices are at distance at most $$2(p+1)$$2(p+1) in an undirected graph, (3) testing if there is a directed path from s to t for two specific vertices s and t in a directed graph. The lower bounds hold for $$p = O(\log n / \log \log n)$$p=O(logn/loglogn). Prior to our result, it was known that these problems require $$\varOmega (n^2)$$Ω(n2) space in one pass, but no $$n^{1+\varOmega (1)}$$n1+Ω(1) lower bound was known for any $$p\ge 2$$p≥2. These streaming results follow from a communication complexity lower bound for a communication game in which the players hold two graphs on the same set of vertices. The task of the players is to find out whether the sets of vertices at distance exactly $$p+1$$p+1 from a specific vertex intersect. The game requires a significant amount of communication only if the players are forced to speak in a specific difficult order. This is reminiscent of lower bounds for communication problems such as indexing and pointer chasing. Among other things, our line of attack requires proving an information cost lower bound for a decision version of the classic pointer chasing problem and a direct sum type theorem for the disjunction of several instances of this problem.

[1]  Ran Raz,et al.  Probabilistic communication complexity of Boolean relations , 1989, 30th Annual Symposium on Foundations of Computer Science.

[2]  Eyal Kushilevitz,et al.  Communication Complexity , 1997, Adv. Comput..

[3]  Leah Epstein,et al.  Improved Approximation Guarantees for Weighted Matching in the Semi-streaming Model , 2009, SIAM J. Discret. Math..

[4]  A. Kemper,et al.  On Graph Problems in a Semi-streaming Model , 2015 .

[5]  A. Razborov Communication Complexity , 2011 .

[6]  Ashish Goel,et al.  On the communication and streaming complexity of maximum bipartite matching , 2012, SODA.

[7]  Graham Cormode,et al.  Robust lower bounds for communication and stream computation , 2008, Theory Comput..

[8]  Andrew McGregor,et al.  Finding Graph Matchings in Data Streams , 2005, APPROX-RANDOM.

[9]  Jaikumar Radhakrishnan,et al.  Privacy and interaction in quantum communication complexity and a theorem about the relative entropy of quantum states , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[10]  Zvi Galil,et al.  Lower bounds on communication complexity , 1984, STOC '84.

[11]  Andrew Chi-Chih Yao,et al.  Informational complexity and the direct sum problem for simultaneous message complexity , 2001, Proceedings 2001 IEEE International Conference on Cluster Computing.

[12]  Mariano Zelke,et al.  Weighted Matching in the Semi-Streaming Model , 2007, Algorithmica.

[13]  Noam Nisan,et al.  Rounds in communication complexity revisited , 1991, STOC '91.

[14]  Ilan Newman,et al.  Private vs. Common Random Bits in Communication Complexity , 1991, Inf. Process. Lett..

[15]  Iordanis Kerenidis,et al.  Lower Bounds on Information Complexity via Zero-Communication Protocols and Applications , 2012, 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science.

[16]  Iordanis Kerenidis,et al.  Lower Bounds on Information Complexity via Zero-Communication Protocols and Applications , 2012, SIAM J. Comput..

[17]  Anand Srivastav,et al.  Bipartite Matching in the Semi-streaming Model , 2012, Algorithmica.

[18]  Ran Raz,et al.  Probabilistic Communication Complexity of Boolean Relations (Extended Abstract) , 1989, IEEE Annual Symposium on Foundations of Computer Science.

[19]  Mikhail Kapralov Improved lower bounds for matchings in the streaming model , 2012, ArXiv.

[20]  Rahul Jain,et al.  A Direct Product Theorem for Two-Party Bounded-Round Public-Coin Communication Complexity , 2012, Algorithmica.

[21]  Claire Mathieu,et al.  Maximum Matching in Semi-streaming with Few Passes , 2011, APPROX-RANDOM.

[22]  Mikhail Kapralov,et al.  Better bounds for matchings in the streaming model , 2012, SODA.

[23]  Sudipto Guha,et al.  Stream Order and Order Statistics: Quantile Estimation in Random-Order Streams , 2009, SIAM J. Comput..

[24]  Joan Feigenbaum,et al.  Graph Distances in the Data-Stream Model , 2008, SIAM J. Comput..

[25]  Mark Braverman,et al.  A Discrepancy Lower Bound for Information Complexity , 2015, Algorithmica.

[26]  Braverman Mark,et al.  A Discrepancy Lower Bound for Information Complexity , 2012 .

[27]  Rahul Jain,et al.  A direct product theorem for bounded-round public-coin randomized communication complexity , 2012, ArXiv.

[28]  Jaikumar Radhakrishnan,et al.  The Communication Complexity of Pointer Chasing , 2001, J. Comput. Syst. Sci..

[29]  Jaikumar Radhakrishnan,et al.  A Direct Sum Theorem in Communication Complexity via Message Compression , 2003, ICALP.

[30]  J. Feigenbaum,et al.  Graph Distances in the DataStream Model , 2015 .

[31]  Sudipto Guha,et al.  Tight Lower Bounds for Multi-pass Stream Computation Via Pass Elimination , 2008, ICALP.

[32]  Ziv Bar-Yossef,et al.  An information statistics approach to data stream and communication complexity , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[33]  Jirí Sgall,et al.  Some bounds on multiparty communication complexity of pointer jumping , 1998, computational complexity.

[34]  Venkatesan Guruswami,et al.  Superlinear Lower Bounds for Multipass Graph Processing , 2013, Computational Complexity Conference.

[35]  Mark Braverman,et al.  Information Equals Amortized Communication , 2011, IEEE Transactions on Information Theory.

[36]  Sreenivas Gollapudi,et al.  Estimating PageRank on graph streams , 2008, PODS.

[37]  Xi Chen,et al.  How to compress interactive communication , 2010, STOC '10.

[38]  Prabhakar Raghavan,et al.  Computing on data streams , 1999, External Memory Algorithms.

[39]  Leah Epstein,et al.  Improved Approximation Guarantees for Weighted Matching in the Semi-Streaming Model , 2010, STACS.

[40]  Mark Braverman Interactive information complexity , 2012, STOC '12.

[41]  Amit Chakrabarti,et al.  Information Complexity versus Corruption and Applications to Orthogonality and Gap-Hamming , 2012, APPROX-RANDOM.

[42]  Sudipto Guha,et al.  Linear programming in the semi-streaming model with application to the maximum matching problem , 2011, Inf. Comput..