System-wide Introspection for Accurate Attribution of Performance Bottlenecks

The architecture of high-end systems is becoming ever more complex with the introduction of multi-chip, multi-core computation nodes. The behavior of shared resources such as caches, memory controllers, and communication interfaces is playing an increasing role in overall system performance. Most current performance tools are oriented toward “firstperson” performance measurement and they are generally characterizing the performance of individual threads based on on-core Hardware Performance Monitoring (HPM). O↵core resources cannot be directly monitored. In contrast, RCRtoolkit is designed to focus on those shared resources by using HPM counters associated with the shared, o↵-core parts of the system. Its resource-centric approach to performance analysis is designed to provide real time introspective feedback to applications and system software. This paper describes experiments with coupling RCRtoolkit to HPCToolkit, a first-person tool that uses event based sampling of on-core events to attribute costs to application using a hierarchical calling context model. The tools augment each other to provide new analysis capabilities and insights not available separately. The structure of RCRtoolkit is presented, as is the integration of the tools. The capabilities of the combined tools are illustrated with several examples (Lattice QCD, Lattice-Boltzmann Magneto-hydrodynamics, FFT). In particular, the combination is used to identify memory bandwidth (as opposed to latency) and memory load balancing problems and to attribute this behavior accurately to specific program constructs.