Topology-Aware MPI Communication and Scheduling for High Performance Computing Systems

Most of the traditional High End Computing (HEC) applications and current petascale applications are written using the Message Passing Interface (MPI) programming model. Consequently, MPI communication primitives (both point to point and collectives) are extensively used across various scientific and HEC applications. The large-scale HEC systems on which these applications run, by necessity, are designed with multiple layers of switches with different topologies like fat-trees (with different kinds of over-subscription), meshes, torus, etc. Hence, the performance of an MPI library, and in turn the applications, is heavily dependent upon how the MPI library has been designed and optimized to take the system architecture (processor, memory, network interface, and network topology) into account. In addition, parallel jobs are typically submitted to such systems through schedulers (such as PBS and SLURM). Currently, most schedulers do not have the intelligence to allocate compute nodes to MPI tasks based on the underlying topology of the system and the communication requirements of the applications. Thus, the performance and scalability of a parallel application can suffer (even using the best MPI library) if topology-aware scheduling is not employed. Moreover, the placement of logical MPI ranks on a supercomputing system can significantly affect overall application performance. A naive task assignment can result in poor locality of communication. Thus, it is important to design optimal mapping schemes with topology information to improve the overall application performance and scalability. It is also critical for users of High Performance Computing