Lower bounds for external memory integer sorting via network coding

Sorting extremely large datasets is a frequently occuring task in practice. These datasets are usually much larger than the computer’s main memory; thus external memory sorting algorithms, first introduced by Aggarwal and Vitter (1988), are often used. The complexity of comparison based external memory sorting has been understood for decades by now, however the situation remains elusive if we assume the keys to be sorted are integers. In internal memory, one can sort a set of n integer keys of Θ(lgn) bits each in O(n) time using the classic Radix Sort algorithm, however in external memory, there are no faster integer sorting algorithms known than the simple comparison based ones. Whether such algorithms exist has remained a central open problem in external memory algorithms for more than three decades. In this paper, we present a tight conditional lower bound on the complexity of external memory sorting of integers. Our lower bound is based on a famous conjecture in network coding by Li and Li (2004), who conjectured that network coding cannot help anything beyond the standard multicommodity flow rate in undirected graphs. The only previous work connecting the Li and Li conjecture to lower bounds for algorithms is due to Adler et al. (2006). Adler et al. indeed obtain relatively simple lower bounds for oblivious algorithms (the memory access pattern is fixed and independent of the input data). Unfortunately obliviousness is a strong limitations, especially for integer sorting: we show that the Li and Li conjecture implies an Ω(n logn) lower bound for internal memory oblivious sorting when the keys are Θ(lgn) bits. This is in sharp contrast to the classic (non-oblivious) Radix Sort algorithm. Indeed going beyond obliviousness is highly non-trivial; we need to introduce several new methods and involved techniques, which are of their own interest, to obtain our tight lower bound for external memory integer sorting.

[1]  Muriel Médard,et al.  An algebraic approach to network coding , 2003, TNET.

[2]  Zongpeng Li,et al.  On achieving optimal throughput with network coding , 2005, Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies..

[3]  Alok Aggarwal,et al.  The input/output complexity of sorting and related problems , 1988, CACM.

[4]  Kasper Green Larsen,et al.  A Faster External Memory Priority Queue with DecreaseKeys , 2019, SODA.

[5]  Rudolf Ahlswede,et al.  Network information flow , 2000, IEEE Trans. Inf. Theory.

[6]  Yijie Han Deterministic sorting in O(nlog log n) time and linear space , 2002, STOC '02.

[7]  Andrew Chi-Chih Yao,et al.  Should tables be sorted? , 1981, 19th Annual Symposium on Foundations of Computer Science (sfcs 1978).

[8]  Elaine Shi,et al.  Lower bounds for external memory integer sorting via network coding , 2020, Commun. ACM.

[9]  Xi Chen,et al.  How to compress interactive communication , 2010, STOC '10.

[10]  Mark Braverman,et al.  Network coding in undirected graphs is either very helpful or not helpful at all , 2017, ITCS.

[11]  Qin Zhang,et al.  The limits of buffering: a tight lower bound for dynamic membership in the external memory model , 2010, STOC '10.

[12]  Shyamkishor Kumar NETWORK CODING THE CASE OF MULTIPLE UNICAST SESSIONS , 2015 .

[13]  Qin Zhang,et al.  On the cell probe complexity of dynamic membership , 2010, SODA '10.

[14]  John Iacono,et al.  Using Hashing to Solve the Dictionary Problem (In External Memory) , 2011, SODA 2011.

[15]  Yijie Han,et al.  Deterministic sorting inO(nlog logn) time and linear space , 2002, STOC 2002.

[16]  Vijay V. Vazirani,et al.  On the capacity of multiple unicast sessions in undirected graphs , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[17]  Baochun Li,et al.  Network Coding : The Case of Multiple Unicast Sessions , 2004 .

[18]  Robert D. Kleinberg,et al.  On the capacity of information networks , 2006, IEEE Transactions on Information Theory.

[19]  Xin Wang,et al.  A reduction approach to the multiple-unicast conjecture in network coding , 2016, 2016 IEEE International Symposium on Information Theory (ISIT).

[20]  Robert D. Kleinberg,et al.  Comparing Network Coding with Multicommodity Flow for the k-pairs Communication Problem , 2004 .

[21]  Yijie Han,et al.  Integer sorting in O(n/spl radic/(log log n)) expected time and linear space , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[22]  Kasper Green Larsen,et al.  DecreaseKeys are expensive for external memory priority queues , 2017, STOC.