A Unified Trace Environment for IBM SP systems

C. Eric Wu, Hubertus Franke, and Yew-Huey Liu IBM T J. Watson Research Center Distributed parallel processing can increase system computing power beyond the limits of current uniprocessor technology. However, programming in such a system based on the message-passing programming model is much more complex than writing sequential programs. To take advantage of the underlying hardware, understanding the communication behavior of parallel programs and system responses to user applications is extremely critical. One common way of monitoring a program’s behavior is to generate trace events while executing the program. Events generated can then be used for other purposes such as debugging and program visualization. However, as we’ll see, such a method potentially requires source code modification, increases overhead, and causes clocksynchronization problems. T o meet these challenges, we developed a Unified Trace Environment for IBM SP systems. The user-level U T E trace libraries require only relinking for generating message-passing and system events. With the UTE, users can generate message-passing events with minimum overhead, and mark specific portions of the program, such as various phases, loops, and routines, for performance analysis and visualization. Most user-level trace tools for messagepassing systems require source code modification to collect message-passing events. More advanced tools such as the Paradyn systeml require no source code modification; they insert the code for performance instrumentation into an application program during execution. However, instrumentation daemons cause substantial overhead. Collecting system events is as important as collecting message-passing events. System and I/O events such as process dispatch and page fault can reveal crucial information on system responses to user applications. The trace facility should also easily expand to trace activities from other software layers, such as parallel I/O file systems and high-level parallel languages. Such expandability enables the same trace facility to trace multiple software systems. One of the most serious problems in trace analysis for distributed parallel systems is clock synchronization. In such a system, multiple processors generate trace records, and often multiple nodes produce separate streams independently. The logical order of events might not be guaranteed in the trace because of discrepancies among local clocks. As a result, many trace facilities must do additional work to ensure consistent time stamps, thus increasing trace overhead. The challenges of trace analysis