Communication latency hiding: model and implementation in high-latency computer networks

The potential of large numbers of workstations for solving very large problems is tremendous Nevertheless it is often considered inappropriate to parallelize applications with a fair amount of communication on computer networks because communication via networks with high latency and low bandwidth presents a technological bottleneck In this paper a model to analyze the gain of communication latency hiding by overlapping computation and communication is described This model captures the limitations and illustrates the opportunities of communication latency hiding for improving speedup and e ciency of parallel computations that can be structured appropriately Fur thermore an implementation of a message passing protocol is presented that incorporates latency hiding on top of the TCP IP transport layer This protocol ensures e cient deadlock free com munication in UNIX network environments Experiments show that the presented latency hiding technique increases the range of applications suited for parallel computing on networks of computers even across the Internet Measurements with a multiprocessor system demonstrate the validity of the latency hiding model for a broader range of parallel architectures Parallel programming with conventional message passing interfaces is only slightly a ected because an additional protocol layer hides the increased complexity from the programmer

[1]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[2]  Kai Hwang,et al.  Advanced computer architecture - parallelism, scalability, programmability , 1992 .

[3]  Laxmi N. Bhuyan,et al.  High-performance computer architecture , 1995, Future Gener. Comput. Syst..

[4]  Peter Arbenz,et al.  SCIDDLE: A tool for large scale distributed computing , 1995, Concurr. Pract. Exp..

[5]  Ian Wakeman,et al.  Is Layering Harmful , 1992 .

[6]  Vaidy S. Sunderam,et al.  PVM: A Framework for Parallel Distributed Computing , 1990, Concurr. Pract. Exp..

[7]  Thomas L. Casavant,et al.  Hardware support for the Seamless programming model , 1992, [Proceedings 1992] The Fourth Symposium on the Frontiers of Massively Parallel Computation.

[8]  Eric de Sturler,et al.  A Performance Model for Krylov Subspace Methods on Mesh-Based Parallel Computers , 1996, Parallel Comput..

[9]  Jehoshua Bruck,et al.  The IBM External User Interface for Scalable Parallel Systems , 1994, Parallel Comput..

[10]  Dan C. Marinescu,et al.  Models and Algorithms for Coscheduling Compute-Intensive Tasks on a Network of Workstations , 1992, J. Parallel Distributed Comput..

[11]  Hans-Christian Hege,et al.  On the Impact of Communication Latencies on Distributed Sparse LU Factorization. , 1993 .

[12]  T. Störtkuhl,et al.  On the Parallel Solution of 3D PDEs on a Network of Workstations and on Vector Computers , 1993, Parallel Computer Architectures.

[13]  Armando P. Stettner.  The design and implementation of the 4.3BSD UNIX operating system , 1988 .

[14]  Volker Strumpen,et al.  Efficient Parallel Computing in Distributed Workstation Environments , 1993, Parallel Comput..

[15]  E. Sturler,et al.  Communication Cost Reduction for Krylov Methods on Parallel Computers , 1994, HPCN.

[16]  Vaidy S. Sunderam,et al.  Superconcurrent simulation of polymer chains on heterogeneous networks , 1992, Proceedings Supercomputing '92.

[17]  Willy Zwaenepoel,et al.  Optimistic implementation of bulk data transfer protocols , 1989, SIGMETRICS '89.

[18]  Peter Arbenz,et al.  Applied Distributed Supercomputing in Homogeneous Networks , 1992, Int. J. High Speed Comput..

[19]  Volker Strumpen.  A Large-Scale Metacomputer Approach for Distributed Parallel Computing , 1994, HPCN.

[20]  Paul Pierce.  The NX Message Passing Interface , 1994, Parallel Comput..

[21]  Message Passing Interface Forum.  MPI: A message - passing interface standard , 1994 .