Enabling Efficient Use of MPI and PGAS Programming Models on Heterogeneous Clusters with High Performance Interconnects

Accelerators (such as NVIDIA GPUs) and coprocessors (such as Intel MIC/Xeon Phi) are fueling the growth of next-generation ultra-scale systems with high compute density and high performance per watt. However, they render these systems heterogeneous by introducing multiple levels of parallelism and varying computation/communication costs at each level. Application developers also use a hierarchy of programming models to extract maximum performance from these heterogeneous systems. They use models like CUDA, OpenCL, LEO+OpenMP, and others on an accelerator or a coprocessor while using higher level programming models like an MPI or a PGAS model across a cluster. Multiple programming models, their runtimes, and varying performance of communication at different levels of the system hierarchy have limited applications from the achieving peak performance on these systems. For example, in MPI and OpenSHMEM applications that run on GPU clusters, data is transferred between GPU and CPU using CUDA while it is exchanged between MPI processes running on different nodes using MPI or OpenSHMEM. This two-stage process for data movement introduces inefficiencies and hence limits performance. Communication in applications running on clusters with Intel MIC can happen over a myriad of channels depending on where the application processes are running: intra-MIC, intra-host, MIC-Host and MIC-MIC. Each of these channels has different performance characteristics. Runtimes have to be re-designed to optimize communication in such scenarios, while hiding system complexity from the user. ii Computation and communication overlap has been been a critical requirement for applications, to achieve peak performance on large-scale systems. Communication overheads have a magnified impact on heterogeneous clusters due to their higher compute density and hence, a higher wastage in compute power. Modern interconnects like InfiniBand, with their Remote DMA capabilities, enable asynchronous progress of communication, freeing up the cores to do useful computation. MPI and PGAS models offer light-weight, one-sided communication primitives that minimize process synchronization overheads and enable better computation and communication overlap. However, the design of one-sided communication on heterogeneous clusters is not well studied. Further, there has been limited literature to guide scientists in taking advantage of the one-sided communication semantics on high-end applications, more so on heterogeneous architectures. This dissertation has targeted several of these challenges for programming on GPU and Intel MIC clusters. Our work with MVAPICH2-GPU enabled the use of MPI in a unified manner, for communication from host and GPU device memories. It takes advantage of unified virtual addressing(UVA) provided by CUDA. We proposed designs in the MVAPICH2-GPU runtime to significantly improve the performance of internode and intranode GPU-GPU communication by pipelining and overlapping memory, PCIe and network transfers. We take advantage of CUDA features, such as IPC, GPUDirect RDMA, and CUDA kernels to further reduce communication overheads. MVAPICH2-GPU improves programmability by removing the need for developers to use CUDA and MPI for GPU-GPU communication, while improving performance through runtime-level optimizations that are transparent to the user. We have shown up to 69% and 45% improvement in point-to-point latency for data movement for 4Byte and 4MB messages, respectively. Likewise, the solutions improve the bandwidth by 2x and 56% for 4KByte and 64 KByte