External memory algorithms

Data sets in large applications are often too massive to fit completely inside the computer's internal memory. The resulting input/ output communication (or I/O) between fast internal memory and slower external memory (such as disks) can be a major performance bottleneck. In this tutorial, we survey the state of the art in the design and analysis of external memory algorithms (also known as EM algorithms or out-of-core algorithms or I/O algorithms). External memory algorithms are often designed using the parallel disk model (PDM). The three machine-independent measures of an algorithm's performance in PDM are the number of I/O operations performed, the CPU time, and the amount of disk space used. PDM allows for multiple disks (or disk arrays) and parallel CPUs, and it can be generalized to handle cache hierarchies, hierarchical memory, and tertiary storage. We discuss a variety of problems and show how to solve them efficiently in external memory. Programming tools and environments are available for simplifying the programming task. Experiments on some newly developed algorithms for spatial databases incorporating these paradigms, implemented using TPIE (Transparent Parallel I/O programming Environment), show significant speedups over popular methods.

[1]  Erik D Vengroff,et al.  I/O Efficient Scientific Computation Using TPIE , 1995 .

[2]  János Komlós,et al.  Hash functions for priority queues , 1983, 24th Annual Symposium on Foundations of Computer Science (sfcs 1983).

[3]  Kenneth L. Clarkson,et al.  Applications of random sampling in computational geometry, II , 1988, SCG '88.

[4]  Yi-Jen Chiang,et al.  Experiments on the Practical I/O Efficiency of Geometric Algorithms: Distribution Sweep vs. Plane Sweep , 1995, WADS.

[5]  David Maier,et al.  Readings in Object-Oriented Database Systems , 1989 .

[6]  Ibrahim Kamel,et al.  Bulk insertion in dynamic r-trees , 1996 .

[7]  Sandeep K. S. Gupta,et al.  Generating Efficient Programs for Two-Level Memories from Tensor-products , 1995, Parallel and Distributed Computing and Systems.

[8]  Donald E. Knuth,et al.  Sorting and Searching , 1973 .

[9]  William Feller,et al.  An Introduction to Probability Theory and Its Applications , 1967 .

[10]  Thomas H. Cormen,et al.  Asymptotically Tight Bounds for Performing BMMC Permutations on Parallel Disk Systems , 1998, SIAM J. Comput..

[11]  Kurt Mehlhorn,et al.  Randomized external-memory algorithms for some geometric problems , 1998, SCG '98.

[12]  Jyh-Jong Tsay,et al.  External-memory computational geometry , 1993, Proceedings of 1993 IEEE 34th Annual Foundations of Computer Science.

[13]  Christos Faloutsos,et al.  On packing R-trees , 1993, CIKM '93.

[14]  Phillip B. Gibbons,et al.  Modeling and optimizing I/O throughput of multiple disks on a bus (summary) , 1998, SIGMETRICS '98/PERFORMANCE '98.

[15]  David G. Kirkpatrick,et al.  The Ultimate Planar Convex Hull Algorithm? , 1986, SIAM J. Comput..

[16]  Frank Dehne,et al.  Efficient External Memory Algorithms by Simulating Coarse-Grained Parallel Algorithms , 1997, SPAA '97.

[17]  Rakesh M. Verma,et al.  An Efficient Multiversion Access STructure , 1997, IEEE Trans. Knowl. Data Eng..

[18]  Joseph W. H. Liu,et al.  On the storage requirement in the out-of-core multifrontal method for sparse factorization , 1986, TOMS.

[19]  Miron Livny,et al.  Memory-Adaptive External Sorting , 1993, VLDB.

[20]  Robert W. Floyd,et al.  Permuting Information in Idealized Two-Level Storage , 1972, Complexity of Computer Computations.

[21]  David S. Greenberg,et al.  Beyond core: Making parallel computer I/O practical , 1993 .

[22]  Derek Thompson,et al.  Fundamentals of spatial information systems , 1992, A.P.I.C. series.

[23]  Peter Widmayer,et al.  Spatial Data Structures: Concepts and Design Choices , 1996, Algorithmic Foundations of Geographic Information Systems.

[24]  Jeffrey Scott Vitter,et al.  Efficient memory access in large-scale computation (invited paper) , 1991 .

[25]  Sridhar Ramaswamy,et al.  Path Caching: A Technique for Optimal External Searching , 1994, PODS 1994.

[26]  Hans-Peter Kriegel,et al.  The R*-tree: an efficient and robust access method for points and rectangles , 1990, SIGMOD '90.

[27]  Jeffrey Scott Vitter,et al.  Efficient searching with linear constraints , 1998, J. Comput. Syst. Sci..

[28]  Robert Laurini,et al.  9 – Design for Information Systems: Methodologies, issues , 1992 .

[29]  Gerth Stølting Brodal,et al.  Worst-Case External-Memory Priority Queues , 1998, SWAT.

[30]  Michael Kaufmann,et al.  BSP-Like External-Memory Computation , 1997, CIAC.

[31]  John H. Reif,et al.  Models and Resource Metrics for Parallel and Distributed Computation , 1994, Parallel Algorithms Appl..

[32]  Gabriel M. Kuper,et al.  Constraint query languages (preliminary report) , 1990, PODS '90.

[33]  Jeffrey Scott Vitter,et al.  Modeling and optimizing I/O throughput of multiple disks on a bus , 1999, SIGMETRICS '99.

[34]  Cláudio T. Silva,et al.  I/O optimal isosurface extraction (extended abstract) , 1997, VIS '97.

[35]  Arif Merchant,et al.  An analytic behavior model for disk drives with readahead caches and request reordering , 1998, SIGMETRICS '98/PERFORMANCE '98.

[36]  Christian Böhm,et al.  Improving the Query Performance of High-Dimensional Index Structures by Bulk-Load Operations , 1998, EDBT.

[37]  Hanan Samet,et al.  Hierarchical Spatial Data Structures , 1989, SSD.

[38]  Sivan Toledo,et al.  A survey of out-of-core algorithms in numerical linear algebra , 1999, External Memory Algorithms.

[39]  Lars Arge,et al.  A General Lower Bound on the I/O-Complexity of Comparison-based Algorithms , 1992, WADS.

[40]  Tse-Yun Feng,et al.  The Universality of the Shuffle-Exchange Network , 1981, IEEE Transactions on Computers.

[41]  Elizabeth Shriver,et al.  An Introduction to Parallel I/O Models and Algorithms , 1996, Input/Output in Parallel and Distributed Computer Systems.

[42]  Edward F. Grove,et al.  External-memory graph algorithms , 1995, SODA '95.

[43]  Sridhar Ramaswamy,et al.  Indexing for data models with constraints and classes (extended abstract) , 1993, PODS '93.

[44]  Diane Greene,et al.  An implementation and performance analysis of spatial data access methods , 1989, [1989] Proceedings. Fifth International Conference on Data Engineering.

[45]  Roberto Grossi,et al.  On sorting strings in external memory (extended abstract) , 1997, STOC '97.

[46]  David Scot Taylor,et al.  Tight bounds for 2-dimensional indexing schemes , 1998, PODS '98.

[47]  Frank Thomson Leighton,et al.  Tight Bounds on the Complexity of Parallel Sorting , 1985, IEEE Trans. Computers.

[48]  David J. DeWitt,et al.  Partition based spatial-merge join , 1996, SIGMOD '96.

[49]  Dror G. Feitelson,et al.  Overview of the MPI-IO Parallel I/O Interface , 1996, Input/Output in Parallel and Distributed Computer Systems.

[50]  Christos H. Papadimitriou,et al.  On the analysis of indexing schemes , 1997, PODS '97.

[51]  Hanan Samet,et al.  The Design and Analysis of Spatial Data Structures , 1989 .

[52]  Garth A. Gibson.  Report of the Working Group on Storage I/O Issues in Large-Scale Computing , 1996 .

[53]  Jeffrey Scott Vitter,et al.  Strategic directions in storage I/O issues in large-scale computing , 1996, CSUR.

[54]  Jeffrey Scott Vitter,et al.  DIMACS workshop on External memory algorithms and visualization , 1999 .

[55]  Sridhar Ramaswamy,et al.  Path caching (extended abstract): a technique for optimal external searching , 1994, PODS '94.

[56]  David M. Nicol,et al.  Performing Out-of Core FFTs on Parallel Disk Systems , 1998, Parallel Comput..

[57]  Micah Adler,et al.  New coding techniques for improved bandwidth utilization , 1996, Proceedings of 37th Conference on Foundations of Computer Science.

[58]  Bernhard Seeger,et al.  An asymptotically optimal multiversion B-tree , 1996, The VLDB Journal.

[59]  Jeffrey Scott Vitter,et al.  Efficient 3-D range searching in external memory , 1996, STOC '96.

[60]  Edward F. Grove,et al.  Simple Randomized Mergesort on Parallel Disks , 1997, Parallel Comput..

[61]  Alok Aggarwal,et al.  The input/output complexity of sorting and related problems , 1988, CACM.

[62]  Oliver Günther,et al.  Multidimensional access methods , 1998, CSUR.

[63]  Sridhar Ramaswamy,et al.  Theory and practice of I/O-efficient algorithms for multidimensional batched searching problems , 1998, SODA '98.

[64]  H. T. Kung,et al.  I/O complexity: The red-blue pebble game , 1981, STOC '81.

[65]  Garth A. Gibson,et al.  RAID: high-performance, reliable secondary storage , 1994, CSUR.

[66]  Jeffrey Scott Vitter,et al.  Scalable mining for classification rules in relational databases , 1998 .

[67]  Bernhard Seeger,et al.  A Generic Approach to Bulk Loading Multidimensional Index Structures , 1997, VLDB.

[68]  Jeffrey Scott Vitter,et al.  Greed sort: optimal deterministic sorting on parallel disks , 1995, JACM.

[69]  Yi-Jen Chiang,et al.  I/O optimal isosurface extraction , 1997, Proceedings. Visualization '97 (Cat. No. 97CB36155).

[70]  Robert E. Tarjan,et al.  Making data structures persistent , 1986, STOC '86.

[71]  David R. Clark,et al.  Efficient suffix trees on secondary storage , 1996, SODA '96.

[72]  Jop F. Sibeyn,et al.  From parallel to external list ranking , 1997 .

[73]  Jeffrey Scott Vitter,et al.  Optimal dynamic interval management in external memory , 1996, Proceedings of 37th Conference on Foundations of Computer Science.

[74]  Lars Arge,et al.  The I/O - Complexity of Ordered Binary - Decision Diagram Manipulation , 1995, ISAAC.

[75]  Lars Arge,et al.  The Buffer Tree: A New Technique for Optimal I/O-Algorithms (Extended Abstract) , 1995, WADS.

[76]  Sivan Toledo,et al.  The design and implementation of SOLAR, a portable library for scalable out-of-core linear algebra computations , 1996, IOPADS '96.

[77]  Roberto Grossi,et al.  Efficient Splitting and Merging Algorithms for Order Decomposable Problems , 1999, Inf. Comput..

[78]  Howard B. Demuth.  Electronic Data Sorting , 1985, IEEE Transactions on Computers.

[79]  Antonin Guttman,et al.  R-trees: a dynamic index structure for spatial searching , 1984, SIGMOD '84.

[80]  Per-Åke Larson,et al.  Dynamic Memory Adjustment for External Mergesort , 1997, VLDB.

[81]  Sridhar Ramaswamy,et al.  Scalable Sweeping-Based Spatial Join , 1998, VLDB.

[82]  Jeffrey Scott Vitter,et al.  Deterministic distribution sort in shared and distributed memory multiprocessors , 1993, SPAA '93.

[83]  Roberto Grossi,et al.  Efficient cross-trees for external memory , 1999, External Memory Algorithms.

[84]  Christos Faloutsos,et al.  Hilbert R-tree: An Improved R-tree using Fractals , 1994, VLDB.

[85]  Donald E. Knuth,et al.  The art of computer programming, volume 3: (2nd ed.) sorting and searching , 1998 .

[86]  Alok Aggarwal,et al.  Optimal parallel sorting in multi-level storage , 1994, SODA '94.

[87]  Vijay Kumar,et al.  Improved algorithms and data structures for solving graph problems in external memory , 1996, Proceedings of SPDP '96: 8th IEEE Symposium on Parallel and Distributed Processing.

[88]  Peter Bro Miltersen,et al.  On showing lower bounds for external-memory computational geometry problems , 1998, External Memory Algorithms.

[89]  David J. DeWitt,et al.  Parallel sorting on a shared-nothing architecture using probabilistic splitting , 1991, [1991] Proceedings of the First International Conference on Parallel and Distributed Information Systems.

[90]  Roberto Grossi,et al.  Efficient Splitting and Merging Algorithms for Order Decomposable Problems (Extended Abstract) , 1997, ICALP.

[91]  Michael T. Goodrich,et al.  Topology B-Trees and Their Applications , 1995, WADS.

[92]  T. M. Murali,et al.  I/O-efficient algorithms for contour-line extraction and planar graph blocking , 1998, SODA '98.

[93]  Ambuj K. Singh,et al.  Optimal Dynamic Range Searching in Non-replicating Index Structures , 1999, ICDT.

[94]  John Wilkes,et al.  An introduction to disk drive modeling , 1994, Computer.

[95]  Jeffrey Scott Vitter,et al.  I/O Overhead and Parallel VLSI Architectures for Lattice Computations , 1991, IEEE Trans. Computers.

[96]  Roberto Grossi,et al.  A fully-dynamic data structure for external substring search , 1995, STOC '95.

[97]  David B. Lomet,et al.  Concurrency and recovery for index trees , 1997, The VLDB Journal.

[98]  C. Greg Plaxton,et al.  Deterministic sorting in nearly logarithmic time on the hypercube and related computers , 1990, STOC '90.

[99]  Jeffrey Scott Vitter,et al.  Efficient Memory Access in Large-Scale Computation , 1991, STACS.

[100]  Roberto Grossi,et al.  Fast string searching in secondary storage: theoretical developments and experimental results , 1996, SODA '96.

[101]  Sridhar Ramaswamy,et al.  Indexing for Data Models with Constraints and Classes , 1996, J. Comput. Syst. Sci..

[102]  Sivan Toledo,et al.  Efficient out-of-core algorithms for linear relaxation using blocking covers , 1993, Proceedings of 1993 IEEE 34th Annual Foundations of Computer Science.