Millennium sort: a cluster-based application for windows NT using DCOM, river primitives and the virtual interface architecture

We present the design and results of Millennium Sort, a distributed sorting application built using three layers of technology: extensible River System primitives, the Virtual Interface Architecture (VIA) and the Distributed Component Object Model (DCOM). The Millennium Sort application is a vehicle for exploring the issues of commercial cluster technologies and distributed development on commodity node clusters. We discuss the architecture and design of the River System primitives, VIA and DCOM. Performance results are discussed, including the latest Datamation Sort record time of 1.18 seconds achieved by a 16-node Pentium-II cluster.

[1]  David E. Culler,et al.  An Implementation and Analysis of the Virtual Interface Architecture , 1998, Proceedings of the IEEE/ACM SC98 Conference.

[2]  Robert Barnes,et al.  Loading databases using dataflow parallelism , 1994, SGMD.

[3]  Jim Gray,et al.  Performance / Price Sort , 1998, ArXiv.

[4]  David E. Culler,et al.  A case for NOW (networks of workstation) , 1995, PODC '95.

[5]  Amin Vahdat,et al.  GLUix: a global layer unix for a network of workstations , 1998 .

[6]  Michael Stonebraker,et al.  A measure of transaction processing power , 1985 .

[7]  Cezary Dubnicki,et al.  VMMC-2 : Efficient Support for Reliable, Connection-Oriented Communication , 1997 .

[8]  Erik Riedel,et al.  A performance study of sequential I/O on windows NT TM 4 , 1998 .

[9]  Erik Riedel,et al.  Sequential I/O on Windows NT™ 4.0 - Achieving Top Performance , 1999 .

[10]  Calton Pu,et al.  Harnessing user-level networking architectures for distributed object computing over high-speed networks , 1998 .

[11]  Charles L. Seitz,et al.  Myrinet: A Gigabit-per-Second Local Area Network , 1995, IEEE Micro.

[12]  Andrea C. Arpaci-Dusseau,et al.  Fast Parallel Sorting Under LogP: Experience with the CM-5 , 1996, IEEE Trans. Parallel Distributed Syst..

[13]  Andrea C. Arpaci-Dusseau,et al.  High-performance sorting on networks of workstations , 1997, SIGMOD '97.

[14]  Greg J. Regnier,et al.  The Virtual Interface Architecture , 2002, IEEE Micro.

[15]  Guy Eddon,et al.  Inside Distributed COM , 1998 .

[16]  Thorsten von Eicken,et al.  Incorporating Memory Management into User-Level Network Interfaces , 1997 .

[17]  Noah Treuhaft,et al.  Cluster I/O with River: making the fast case common , 1999, IOPADS '99.