Scalable Job Startup and Inter-Node Communication in Multi-Core InfiniBand Clusters

Moores Law – frequency scaling and exploitation of Instruction Level Parallelism due to increasing transistor density no longer leads performance gains in modern systems due to limitations in power dissipation. This has led to an increased focus on deriving performance gains by taking advantage of Data Level Parallelism through parallel computing. Clusters – groups of commodity compute nodes connected via a modern interconnect have emerged as the top supercomputers in the World and the Message Passing Interface (MPI) has emerged as the de facto standard in parallel processing models on large clusters. Scientific and financial applications have ever increasing demands for compute cycles and the emergence of multi-core processors has driven an enormous growth in the cluster sizes in recent years. InfiniBand has emerged as a popular low-latency, high-bandwidth interconnect of choice in these large clusters. With cluster sizes continuing to scale, the scalability of MPI libraries and associated system support and resources such as the job launcher have been at the center of attention in the High Performance Computing (HPC) community. In this work we examine the current job launching mechanisms that have scalability problems on large scale clusters due to resource constraints as well as performance bottlenecks. We propose a Scalable and Extensible Launching Architecture for Clusters (ScELA) that scales to modern clusters such as the 64K processor TACC Ranger. ii We also examine the scalability constraints with point-to-point InfiniBand channels in MPI libraries. We use the eXtended Reliable Connection (XRC) transport available in recent InfiniBand adapters to design a scalable MPI communication channel with a smaller memory footprint. The designs proposed in this work are available in both MVAPICH and MVAPICH2 MPI libraries over InfiniBand, which are used by more than nine hundred organizations around the World.