Using a virtual event space to understand parallel application communication behavior

For scientific application run on clusters, communication performance becomes increasingly important when the number of cluster nodes increase. To understand the communication behavior, we have developed EventSpace, a configurable data collecting, management and observation system for monitoring low-level synchronization and communication events. Applications are instrumented by adding data collecting code in the form of event collectors to an applications communication paths. When triggered these create and store virtual events to a virtual event space. Based on the meta-data describing the communication paths, virtual events can be combined to provide different views of the applications communication behavior. We used the data collected by EventSpace to do a post-mortem analysis of a wind-tunnel application, a river simulator, global clock synchronization, and a collective operation. The views allowed us to detect anomalous communication behavior, detect load balance problems, find hotspots in a collective communication structure, synchronize the Pentium timestamp counters on the cluster nodes, and analyze the accuracy of the synchronization.