Cluster I/O with River: making the fast case common

We introduce River, a data-flow programming environment and I/O substrate for clusters of computers. River is designed to provide maximum performance in the common case — even in the face of nonuniformities in hardware, software, and workload. River is based on two simple design features: a high-performance distributed queue, and a storage redundancy mechanism called graduated declustering. We have implemented a number of data-intensive applications on River, which validate our design with near-ideal performance in a variety of non-uniform performance scenarios.

[1]  Fritz Leiber,et al.  The Big Time , 1958 .

[2]  E. F. CODD,et al.  A relational model of data for large shared data banks , 1970, CACM.

[3]  E. F. Codd,et al.  A relational model of data for large shared data banks , 1970, CACM.

[4]  Tanaka Hidehiko,et al.  GRACE: Relational Algebra Machine Based on Hash and Sort : Its Design Concepts , 1983 .

[5]  Jerome H. Saltzer,et al.  End-to-end arguments in system design , 1984, TOCS.

[6]  Nicholas Carriero,et al.  Parallel Programming in Linda , 1985, ICPP.

[7]  Leonard D. Shapiro,et al.  Join processing in database systems with large main memories , 1986, TODS.

[8]  Jr. Nicholas John Carriero.  Implementation of tuple space machines , 1987 .

[9]  David J. DeWitt,et al.  A performance analysis of the gamma database machine , 1988, SIGMOD '88.

[10]  Tom W. Keller,et al.  Data placement in Bubba , 1988, SIGMOD '88.

[11]  Dennis M. Ritchie,et al.  A stream input-output system , 1990 .

[12]  Goetz Graefe,et al.  Encapsulation of parallelism in the Volcano query processing system , 1990, SIGMOD '90.

[13]  David J. DeWitt,et al.  Chained declustering: a new availability strategy for multiprocessor database machines , 1990, [1990] Proceedings. Sixth International Conference on Data Engineering.

[14]  Darrell D. E. Long,et al.  Swift: Using Distributed Disk Striping to Provide High I/O Data Rates , 1991, Comput. Syst..

[15]  Henri E. Bal,et al.  Orca: A Language For Parallel Programming of Distributed Systems , 1992, IEEE Trans. Software Eng..

[16]  David J. DeWitt,et al.  Parallel database systems: the future of high performance database systems , 1992, CACM.

[17]  S. Kleiman,et al.  Symmetric multiprocessing in Solaris 2.0 , 1992, Digest of Papers COMPCON Spring 1992.

[18]  Seth Copen Goldstein,et al.  Active Messages: A Mechanism for Integrated Communication and Computation , 1992, [1992] Proceedings the 19th Annual International Symposium on Computer Architecture.

[19]  T. von Eicken,et al.  Parallel programming in Split-C , 1993, Supercomputing '93.

[20]  Richard Wheeler,et al.  it/sfs: A Parallel File System for the CM-5 , 1993, USENIX Summer.

[21]  Goetz Graefe,et al.  Algebraic Optimization of Computations over Scientific Databases , 1993, IEEE Data Eng. Bull..

[22]  Andrea C. Arpaci-Dusseau,et al.  Parallel programming in Split-C , 1993, Supercomputing '93. Proceedings.

[23]  Michael Stonebraker,et al.  Tioga: Providing Data Management Support for Scientific Visualization Applications , 1993, VLDB.

[24]  Corporate The MPI Forum,et al.  MPI: a message passing interface , 1993, Supercomputing '93.

[25]  G. A. Geist,et al.  The evolution of the PVM concurrent computing system , 1993, Digest of Papers. Compcon Spring.

[26]  Robert Barnes,et al.  Loading databases using dataflow parallelism , 1994, SGMD.

[27]  Rakesh Krishnaiyer,et al.  PASSION: Parallel And Scalable Software for Input-Output , 1994 .

[28]  David Kotz,et al.  Disk-directed I/O for MIMD multiprocessors , 1994, OSDI '94.

[29]  David J. DeWitt,et al.  Operating System Support for High Performance Parallel I/O Systems , 1994 .

[30]  Charles L. Seitz,et al.  Myrinet: A Gigabit-per-Second Local Area Network , 1995, IEEE Micro.

[31]  Andrew A. Chien,et al.  PPFS: a high performance portable parallel file system , 1995, ICS '95.

[32]  David E. Culler,et al.  Active message applications programming interface and communication subsystem organization , 1995 .

[33]  Scott Pakin,et al.  High Performance Messaging on Workstations: Illinois Fast Messages (FM) for Myrinet , 1995, Proceedings of the IEEE/ACM SC95 Conference.

[34]  Katherine Yelick,et al.  Multipol: A Distributed Data Structure Library , 1995 .

[35]  Thorsten von Eicken,et al.  U-Net: a user-level network interface for parallel and distributed computing , 1995, SOSP.

[36]  David E. Culler,et al.  Active Message Applications Programming Interface , 1996 .

[37]  Monica S. Lam,et al.  Transparent Fault Tolerance for Parallel Applications on Networks of Workstations , 1996, USENIX Annual Technical Conference.

[38]  Dror G. Feitelson,et al.  The Vesta parallel file system , 1996, TOCS.

[39]  Seth Copen Goldstein,et al.  Lazy Threads: Implementing a Fast Parallel Call , 1996, J. Parallel Distributed Comput..

[40]  Bradley C. Kuszmaul,et al.  Cilk: an efficient multithreaded runtime system , 1995, PPOPP '95.

[41]  R. V. Meter.  Observing the effects of multi-zone disks , 1997 .

[42]  Margo I. Seltzer,et al.  Self-monitoring and self-adapting operating systems , 1997, Proceedings. The Sixth Workshop on Hot Topics in Operating Systems (Cat. No.97TB100133).

[43]  David Kotz,et al.  The galley parallel file system , 1997, ICS '96.

[44]  Robert D. Blumofe,et al.  Adaptive and Reliable ParallelComputing9 Networks of Workstations , 1997 .

[45]  Margo Seltzer,et al.  Self-monitoring and self- adapting systems , 1997 .

[46]  Jeanna Neefe Matthews,et al.  Improving the performance of log-structured file systems with adaptive methods , 1997, SOSP.

[47]  Andrea C. Arpaci-Dusseau,et al.  High-performance sorting on networks of workstations , 1997, SIGMOD '97.

[48]  Peter Reading,et al.  High-throughput deep sequencing reveals that microRNAs play important roles in salt tolerance of euhalophyte Salicornia europaea , 2015, BMC Plant Biology.

[49]  Andrea C. Arpaci-Dusseau,et al.  Searching for the sorting record: experiences in tuning NOW-Sort , 1998, SPDT '98.

[50]  Steve Kubica,et al.  Data Parallel Programming with the Khoros Data Services Library , 1998, IPPS/SPDP Workshops.