Exploring the relationship between parallel application run-time and network performance in clusters

Highly variable parallel application execution time is a persistent issue in cluster computing environments, and can be particularly acute in systems composed of networks of workstations (NOWs). We are looking at this issue in terms of consistency. In particular, we are focusing on network performance. Before we can use techniques from fault management to attain consistency, this paper presents our preliminary analysis of run-time variability from logs and experiments, exposing important issues related to systemic inconsistency in NOW clusters. The characterization of application sensitivity can be used to set network performance goals, thereby defining operational requirements. Network performance depends on the virtual topology imposed by the scheduler's allocation of nodes and the communication patterns of the set of running applications. Therefore it is important to look at both the network and the cluster's centralized node mapper (scheduler) as critical subsystems.

[1]  Duncan A. Grove,et al.  Precise MPI Performance Measurement Using MPIBench , 2001 .

[2]  Barton P. Miller,et al.  Using Dynamic Kernel Instrumentation for Kernel and Application Tuning , 1999, Int. J. High Perform. Comput. Appl..

[3]  Jens Mache,et al.  Job scheduling that minimizes network contention due to both communication and I/O , 2000, Proceedings 14th International Parallel and Distributed Processing Symposium. IPDPS 2000.

[4]  I-Hsin Chung,et al.  Active Harmony: Towards Automated Performance Tuning , 2002, ACM/IEEE SC 2002 Conference (SC'02).

[5]  Ralf H. Reussner,et al.  SKaMPI: A Detailed, Accurate MPI Benchmark , 1998, PVM/MPI.

[6]  Thomas Sterling.  Chiba City: The Argonne Scalable Cluster , 2001 .

[7]  Remy Evard.  Chiba city: the Argonne scalable cluster , 2001 .

[8]  Michael Mikolajczak,et al.  Designing And Building Parallel Programs: Concepts And Tools For Parallel Software Engineering , 1997, IEEE Concurrency.

[9]  PredictionCelso L. Mendes,et al.  Performance Stability and Prediction , 1994 .

[10]  Sivarama P. Dandamudi,et al.  Reducing hot-spot contention in shared-memory multiprocessor systems , 1999, IEEE Concurr..

[11]  Pedro López,et al.  A first implementation of in-transit buffers on myrinet gm software , 2001, Proceedings 15th International Parallel and Distributed Processing Symposium. IPDPS 2001.

[12]  Fabrizio Petrini,et al.  Using multirail networks in high-performance clusters , 2001, Proceedings 42nd IEEE Symposium on Foundations of Computer Science.

[13]  William Gropp,et al.  Reproducible Measurements of MPI Performance Characteristics , 1999, PVM/MPI.

[14]  Mark J. Clement,et al.  Core Algorithms of the Maui Scheduler , 2001, JSSPP.

[15]  Jeffrey S. Vetter,et al.  Real-Time Performance Monitoring, Adaptive Control, and Interactive Steering of Computational Grids , 2000, Int. J. High Perform. Comput. Appl..

[16]  Jack Dongarra,et al.  PVM: Parallel virtual machine: a users' guide and tutorial for networked parallel computing , 1995 .

[17]  Pedro López,et al.  A congestion control mechanism for wormhole networks , 2001, Proceedings Ninth Euromicro Workshop on Parallel and Distributed Processing.

[18]  D.E. Culler,et al.  Effects Of Communication Latency, Overhead, And Bandwidth In A Cluster Architecture , 1997, Conference Proceedings. The 24th Annual International Symposium on Computer Architecture.

[19]  William Gropp,et al.  Prototype of AM3: active mapper and monitoring module for Myrinet environments , 2002, 27th Annual IEEE Conference on Local Computer Networks, 2002. Proceedings. LCN 2002..

[20]  Anthony Skjellum,et al.  Using MPI - portable parallel programming with the message-parsing interface , 1994 .

[21]  Zhou Lei,et al.  The portable batch scheduler and the maui scheduler on linux clusters , 2000 .

[22]  David E. Culler,et al.  A case for NOW (networks of workstation) , 1995, PODC '95.

[23]  Karsten Schwan,et al.  Progress: A Toolkit for Interactive Program Steering , 1995, ICPP.

[24]  Michael Jurczyk.  TRAFFIC CONTROL IN WORMHOLE-ROUTING MULTISTAGE INTERCONNECTION NETWORKS , 2000 .

[25]  Ramesh Subramonian,et al.  LogP: towards a realistic model of parallel computation , 1993, PPOPP '93.

[26]  William Gropp.  An introduction to performance debugging for parallel computers , 1997 .

[27]  Jeffrey K. Hollingsworth,et al.  Exposing application alternatives , 1999, Proceedings. 19th IEEE International Conference on Distributed Computing Systems (Cat. No.99CB37003).

[28]  David A. Patterson,et al.  A case for networks of workstations (now) , 1994, Symposium Record Hot Interconnects II.

[29]  Xiaodong Zhang,et al.  Coordinating Parallel Processes on Networks of Workstations , 1997, J. Parallel Distributed Comput..

[30]  Karsten Schwan,et al.  Falcon: On-line Monitoring and Steering of Parallel Programs , 1995 .

[31]  Dror G. Feitelson,et al.  Utilization and Predictability in Scheduling the IBM SP2 with Backfilling , 1998, Proceedings of the First Merged International Parallel Processing Symposium and Symposium on Parallel and Distributed Processing.

[32]  David A. Lifka,et al.  The ANL/IBM SP Scheduling System , 1995, JSSPP.

[33]  Thomas R. Gross,et al.  Direct queries for discovering network resource properties in a distributed environment , 1999, Proceedings. The Eighth International Symposium on High Performance Distributed Computing (Cat. No.99TH8469).

[34]  Cho-Li Wang,et al.  Contention-free complete exchange algorithm on clusters , 2000, Proceedings IEEE International Conference on Cluster Computing. CLUSTER 2000.

[35]  Ronald Minnich,et al.  Supermon: a high-speed cluster monitoring system , 2002, Proceedings. IEEE International Conference on Cluster Computing.

[36]  Anthony Skjellum,et al.  A Fine-Grain Clock Synchronization Mechanism for QoS Based Communication on Myrinet , 2000 .

[37]  P. Sadayappan,et al.  Selective buddy allocation for scheduling parallel jobs on clusters , 2002, Proceedings. IEEE International Conference on Cluster Computing.