Multiprocessor performance debugging and memory bottlenecks

Driven by the computational demands of scientists and engineers, computer architects are building increasingly complex multiprocessor systems. However, while the peak Gigaflop ratings of such systems are often impressive, the actual performance of initial implementations of applications can be disappointing. To make the task of performance debugging manageable, tools are needed that can analyze program behavior and report sources of performance loss. This dissertation describes techniques for building such tools for shared memory multiprocessors. Previous efforts to build performance debugging systems for shared memory multiprocessors had two shortcomings. First, though memory hierarchy performance is often critical to whole program performance, most tools cannot distinguish the time the CPU is computing from the time when it is stalled waiting on the memory hierarchy. Second, many tools significantly perturb a program's execution adding 50% or more overhead, making it difficult to measure the behavior of the original uninstrumented code. This dissertation addresses both of these problems. It describes a software instrumentation system, Mtool, that typically increases program execution time by less than 10% while collecting a detailed profile of where processors are doing work, waiting for work, or stalled waiting on the memory hierarchy. A window-based user interface allows the user to interpret the profile, viewing compute, memory, and synchronization bottlenecks at increasing levels of detail, from a whole program level down to the level of individual procedures, loops, and synchronization objects. In addition to introducing Mtool, we present extensive data on the overhead of collecting basic block count profiles, provide a characterization of the memory overheads in the SPEC benchmark suite, and offer several case studies to illustrate how the features of Mtool have helped users to improve multiprocessor program performance.