Building a parallel pipelined external memory algorithm library

Large and fast hard disks for little money have enabled the processing of huge amounts of data on a single machine. For this purpose, the well-established STXXL library provides a framework for external memory algorithms with an easy-to-use interface. However, the clock speed of processors cannot keep up with the increasing bandwidth of parallel disks, making many algorithms actually compute-bound.

[1]  Jeffrey Scott Vitter,et al.  Algorithms for parallel memory, II: Hierarchical multilevel memories , 1992, Algorithmica.

[2]  Edward F. Grove,et al.  External-memory graph algorithms , 1995, SODA '95.

[3]  Peter Sanders,et al.  MCSTL: The Multi-core Standard Template Library , 2007, Euro-Par.

[4]  Peter Sanders,et al.  Better external memory suffix array construction , 2008, JEAL.

[5]  Lars Arge,et al.  On external-memory MST, SSSP and multi-way planar graph separation , 2004, J. Algorithms.

[6]  Alexander A. Stepanov,et al.  C++ Standard Template Library , 2000 .

[7]  Thomas H. Cormen,et al.  Building on a framework: using FG for more flexibility and improved performance in parallel programs , 2005, 19th IEEE International Parallel and Distributed Processing Symposium.

[8]  Thomas H. Cormen,et al.  FG: A Framework Generator for Hiding Latency in Parallel Programs Running on Clusters , 2004, ISCA PDCS.

[9]  Peter Sanders,et al.  Asynchronous parallel disk sorting , 2003, SPAA '03.

[10]  Roman Dementiev,et al.  Algorithm engineering for large data sets , 2007 .

[11]  Lars Arge,et al.  The Buuer Tree: a New Technique for Optimal I/o-algorithms ? , 1995 .

[12]  Peter Sanders.  Fast Priority Queues for Cached Memory , 1999, ALENEX.

[13]  S. VitterJ.,et al.  Algorithms for parallel memory, I , 1994 .

[14]  Peter Sanders,et al.  STXXL: standard template library for XXL data sets , 2008, Softw. Pract. Exp..

[15]  Lars Arge,et al.  On external-memory MST, SSSP and multi-way planar graph separation , 2000, J. Algorithms.

[16]  Jeffrey Scott Vitter,et al.  Algorithms for parallel memory, I: Two-level memories , 2005, Algorithmica.