Characterizing CUDA Unified Memory (UM)-Aware MPI Designs on Modern GPU Architectures

The CUDA Unified Memory (UM) interface enables a significantly simpler programming paradigm and has the potential to fundamentally change the way programmers write CUDA applications in the future. Although UM leads to high productivity in programming using CUDA by simplifying the programmer's view of CPU and GPU memory spaces, initial support for UM in the Kepler series of GPUs lacked in performance necessitating several UM-aware designs in state-of-the-art MPI runtimes such as MVAPICH2-GDR. This has enabled end MPI applications to take advantage of the high productivity promised by UM along with high performance. However, as CUDA runtimes and GPU architectures advance, the performance offered by UM has also improved significantly. Thus, there is a need to re-evaluate the performance characteristics of UM in light of these changes to understand how the UM-aware designs in state-of-the-art MPI runtimes must be adapted. We take up this broad challenge and characterize the performance of UM-aware MPI operations and to gain insights on how MPI runtimes need to deal with UM-based data residing on GPU and CPU for different generations of GPU architectures. Our characterization studies show that UM designs conceived during the Kepler GPU era still stands valid and provide valuable performance improvement on the latest Pascal and Volta GPUs. Furthermore, performance evaluation of optimized UM designs show that they outperform naive designs on MVAPICH2-GDR and Open MPI by 4.2x and 2.8x respectively for Intel systems. Additionally, the DD experiments for pure device transfers also show that MVAPICH2-GDR is up to 12.6x better than OpenMPI (w/ UCX).