Analysis of Statistical Sampling in Microarchitecture Simulation: Metric, Methodology and Program Characterization

Statistical sampling, especially stratified random sampling, is a promising technique for estimating the performance of the benchmark program without executing the complete program on microarchitecture simulators or real machines. The accuracy of the performance estimate and the simulation cost depend on the three parameters, namely the interval size, the sample size, and the number of phases (or strata). Optimum values for these three parameters depends on the performance behavior of the program and the microarchitecture configuration being evaluated. In this paper, we quantify the effect of these three parameters and their interactions on the accuracy of the performance estimate and simulation cost. We use the Confidence Interval of estimated Mean (CIM), a metric derived from statistical sampling theory, to measure the accuracy of the performance estimate; we also discuss why CIM is an appropriate metric for this analysis. We use the total number of instructions simulated and the total number of samples measured as cost parameters. Finally, we characterize 21 SPEC CPU2000 benchmarks based on our analysis.

[1]  Thomas F. Wenisch,et al.  SMARTS: accelerating microarchitecture simulation via rigorous statistical sampling , 2003, ISCA '03.

[2]  David J. Lilja,et al.  Dynamic Code Region (DCR) Based Program Phase Tracking and Prediction for Dynamic Optimizations , 2005, HiPEAC.

[3]  Tao Li,et al.  Complexity-based program phase analysis and classification , 2006, 2006 International Conference on Parallel Architectures and Compilation Techniques (PACT).

[4]  David J. Lilja,et al.  TR 05-044 PASS : Program Structure Aware Stratified Sampling for Statistically Selecting Instruction Traces and Simulation Points , 2005 .

[5]  Douglas M. Hawkins,et al.  CIM: A Reliable Metric for Evaluating Program Phase Classifications , 2007, IEEE Computer Architecture Letters.

[6]  Brad Calder,et al.  Automatically characterizing large scale program behavior , 2002, ASPLOS X.

[7]  Thomas M. Conte,et al.  Reducing state loss for effective trace sampling of superscalar processors , 1996, Proceedings International Conference on Computer Design. VLSI in Computers and Processors.

[8]  Brad Calder,et al.  Motivation for Variable Length Intervals and Hierarchical Phase Behavior , 2005, IEEE International Symposium on Performance Analysis of Systems and Software, 2005. ISPASS 2005..

[9]  Thomas F. Wenisch,et al.  An Evaluation of Stratified Sampling of Microarchitecture Simulations , 2004 .

[10]  A. Winsor.  Sampling techniques. , 2000, Nursing times.

[11]  Wei Liu,et al.  EXPERT: expedited simulation exploiting program behavior repetition , 2004, ICS '04.

[12]  Ryan N. Rakvic,et al.  The Fuzzy Correlation between Code and Performance Predictability , 2004, 37th International Symposium on Microarchitecture (MICRO-37'04).

[13]  Timothy Sherwood,et al.  Wavelet-based phase classification , 2006, 2006 International Conference on Parallel Architectures and Compilation Techniques (PACT).

[14]  Brad Calder,et al.  Basic block distribution analysis to find periodic behavior and simulation points in applications , 2001, Proceedings 2001 International Conference on Parallel Architectures and Compilation Techniques.

[15]  Thomas M. Conte,et al.  Combining Trace Sampling with Single Pass Methods for Efficient Cache Simulation , 1998, IEEE Trans. Computers.