Automatic MPI Counter Profiling of All Users: First Results on a CRAY T3E 900-512

This paper presents an automatic counter instrumentation and profiling module added to the MPI library on our Cray T3E. A statistical summary of the MPI calls of any MPI partition is gathered during execution and written in MPI Finalize on a special syslog file. The user can get the same statistical information on a file. Weekly and monthly a statistical summary is computed and the user specific part is sent by mail to each user. A summary of 6 month is presented in this paper. It is the first time that all MPI applications on such a large system where automatically instrumented and profiled for such a period. The statistics give new insight in how efficiently the MPP system is really used by the MPI applications. Moreover, it gives hints which application and which MPI routine should be optimized. The software is portable to other systems.