Sublinear Algorithms in the External Memory Model

We initiate the study of sublinear-time algorithms in the external memory model. In this model, the data is stored in blocks of a certain size B, and the algorithm is charged a unit cost for each block access. This model is well-studied, since it reflects the computational issues occurring when the (massive) input is stored on a disk. Since each block access operates on B data elements in parallel, many problems have external memory algorithms whose number of block accesses is only a small fraction (e.g. 1/B) of their main memory complexity. However, to the best of our knowledge, no such reduction in complexity is known for any sublinear-time algorithm. One plausible explanation is that the vast majority of sublinear-time algorithms use random sampling and thus exhibit no locality of reference. This state of affairs is quite unfortunate, since both sublinear-time algorithms and the external memory model are important approaches to dealing with massive data sets, and ideally they should be combined to achieve best performance. We show that such combination is indeed possible. In particular, we consider three well-studied problems: testing of distinctness, uniformity and identity of an empirical distribution induced by data. For these problems we show random-sampling-based algorithms whose number of block accesses is up to a factor of 1/√B smaller than the main memory complexity of those problems.We also show that this improvement is optimal for those problems. Since these problems are natural primitives for a number of samplingbased algorithms for other problems, our tools improve the external memory complexity of other problems as well.

[1]  Frank Olken,et al.  Random Sampling from Databases , 1993 .

[2]  VitterJeffrey Scott External memory algorithms and data structures , 2001 .

[3]  Oded Goldreich,et al.  Combinatorial property testing (a survey) , 1997, Randomization Methods in Algorithm Design.

[4]  Jeffrey Scott Vitter,et al.  External memory algorithms and data structures: dealing with massive data , 2001, CSUR.

[5]  Dana Ron,et al.  On Testing Expansion in Bounded-Degree Graphs , 2000, Studies in Complexity and Cryptography.

[6]  Doron Rotem,et al.  Simple Random Sampling from Relational Databases , 1986, VLDB.

[7]  Jeffrey Scott Vitter,et al.  External memory algorithms and data structures , 1999, External Memory Algorithms.

[8]  Eldar Fischer,et al.  A Review of Graph Grammars and Preview of ICGT 2002: The First International Conference on Graph Transformation. , 2001 .

[9]  Ronitt Rubinfeld,et al.  Testing that distributions are close , 2000, Proceedings 41st Annual Symposium on Foundations of Computer Science.

[10]  Robin Milner,et al.  On Observing Nondeterminism and Concurrency , 1980, ICALP.

[11]  Ravi Kumar,et al.  Sampling algorithms: lower bounds and applications , 2001, STOC '01.

[12]  Ronitt Rubinfeld,et al.  Testing random variables for independence and identity , 2001, Proceedings 2001 IEEE International Conference on Cluster Computing.

[13]  E. Fischer THE ART OF UNINFORMED DECISIONS: A PRIMER TO PROPERTY TESTING , 2004 .

[14]  Krzysztof Onak,et al.  Testing Properties of Sets of Points in Metric Spaces , 2008, ICALP.

[15]  Eldar Fischer,et al.  Testing graph isomorphism , 2006, SODA '06.

[16]  Tugkan Batu Testing Properties of Distributions , 2001 .