Evaluation of Energy Characteristics of MPI Communication Primitives with RAPL

The energy consumed by modern supercomputing systems continues to grow at an alarming rate. The Message Passing Interface (MPI) has been the de facto programming model for parallel applications and MPI libraries have been designed to achieve the best communication performance on modern architectures. However, the performance and energy trade-offs of these designs have not been studied. Hence, it is critical to understand the energy consumption characteristics of MPI routines and the performance-energy trade-offs of various protocols and designs that are used in MPI libraries. The first hurdle in achieving this objective is to design a framework that can be used to measure energy consumption of various components during communication operations. The RAPL interface allows users to measure energy across various domains on the Intel Sandy-Bridge processor, in a low-overhead, non-intrusive manner. However, this interface has certain limitations and cannot be directly used to measure energy profiles of MPI operations in a fine-grained manner. In this paper, we propose a novel methodology to address these limitations. We propose a new shared-memory window-based solution to accurately measure the aggregate energy consumed by all processes engaged in MPI operations. Using our proposed framework, we demonstrate the impact of various communication protocols and progress mechanisms on the energy consumption. Our evaluations demonstrate that the kernel-based solutions can potentially lead to lower energy consumption for intra-node communication operations. Further, our framework also reveals possible energy bottlenecks in scaling important collective operations, such as, MPI All reduce. In addition, we also use our proposed framework to study the energy consumption characteristics of MPI calls in the NAS-IS benchmark and we infer that the choice of progress mechanism can lead to about 6% energy savings for the processors.