Runtime variability in scientific parallel applications

Simulation remains an important component in the design of multicore processor architectures, just as in uniprocessor design. In contrast to single-threaded applications, however, many multi-threaded programs are not deterministic: in multiple runs, even on the same architecture, different execution paths can be taken. This results in variability of simulation results, which is a well-known phenomenon in real systems, but is nearly universally ignored in simulation experiments. In this paper, we review existing work on simulation variability. We extend this work, which has been focused mainly on commercial workloads, and show that it also applies to scientific workloads. We characterize variability for the SPLASH-2 benchmark applications, and look at how variability can impact the optimization of the interconnection network. Both previous and our own results show that studies aiming to prove the optimality of architectural or other modifications should keep this variability in mind, and make sure that observed improvements are statistically valid in relation to the inherent variability to avoid drawing the wrong conclusions. Although this problem is in no way solved to satisfaction, we review some possible solutions, such as the use of statistics, using different types of metrics, and sample-based simulation.