Characterization of SPEC CPU2006 and SPEC OMP2001: Regression Models and their Transferability

Analysis of workload execution and identification of software and hardware performance barriers provide critical engineering benefits; these include guidance on software optimization, hardware design tradeoffs, configuration tuning, and comparative assessments for platform selection. This paper uses Model trees to build statistical regression models for the SPEC1 CPU2006 and the SPEC OMP2001 suites. These models link performance to key microarchitectural events. The models provide detailed recipes for identifying the key performance factors for each suite and for determining the contribution of each factor to performance. The paper discusses how the models can be used to understand the behaviors of the two suites on a modern processor. These models are applied to obtain a detailed performance characterization of each benchmark suite and its member workloads and to identify the commonalities and distinctions among the performance factors that affect each of the member workloads within the two suites. This paper also addresses the issue of model transferability. It explores the question: How useful is an existing performance model (built on a given suite of workloads) to study the performance of different workloads or suites of workloads? A performance model built using data from workload suite P is considered transferable to workload suite Q if it can be used to accurately study the performance of workload suite Q. Statistical methodologies to assess model transferability are discussed. In particular, the paper explores the use of two-sample hypothesis tests and prediction accuracy analysis techniques to assess model transferability. It is found that a model trained using only 10% of the SPEC CPU2006 data is transferable to the remaining data. This finding holds also for SPEC OMP2001. In contrast, it is found that the SPEC CPU2006 model is not transferable to SPEC OMP2001 and vice versa.

[1]  John L. Henning.  SPEC CPU2006 memory footprint , 2007, CARN.

[2]  Rudolf Eigenmann,et al.  Quantitative performance analysis of the SPEC OMPM2001 benchmarks , 2003, Sci. Program..

[3]  John L. Henning,et al.  Subroutine profiling results for the CPU2006 benchmarks , 2007, CARN.

[4]  Rudolf Eigenmann,et al.  Large System Performance of SPEC OMP2001 Benchmarks , 2002, ISHPC.

[5]  Benjamin C. Lee.  An Architectural Assessment of SPEC CPU Benchmark Relevance , 2006 .

[6]  Rudolf Eigenmann,et al.  SPEComp: A New Benchmark Suite for Measuring Parallel Computer Performance , 2001, WOMPAT.

[7]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques with Java implementations , 2002, SGMD.

[8]  Sally A. McKee,et al.  Methods of inference and learning for performance modeling of parallel applications , 2007, PPoPP.

[9]  Ian Witten,et al.  Data Mining , 2000 .

[10]  J. R. Quinlan.  Learning With Continuous Classes , 1992 .

[11]  Rudolf Eigenmann,et al.  Performance characteristics of the SPEC OMP2001 benchmarks , 2001, CARN.

[12]  Charles Yount,et al.  On the Comparison of Regression Algorithms for Computer Architecture Performance Analysis of Software Applications , .

[13]  D.J. Lilja,et al.  Accurate statistical approaches for generating representative workload compositions , 2005, IEEE International. 2005 Proceedings of the IEEE Workload Characterization Symposium, 2005..

[14]  L. Eeckhout,et al.  Exploiting program microarchitecture independent characteristics and phase behavior for reduced benchmark suite simulation , 2005, IEEE International. 2005 Proceedings of the IEEE Workload Characterization Symposium, 2005..

[15]  Matthias S. Müller.  SPEC OpenMP Benchmarks on Four Generations of NEC SX Parallel Vector Systems , 2005, IWOMP.

[16]  Lieven Eeckhout,et al.  Evaluating Benchmark Subsetting Approaches , 2006, 2006 IEEE International Symposium on Workload Characterization.

[17]  Lizy Kurian John,et al.  Analysis of redundancy and application balance in the SPEC CPU2006 benchmark suite , 2007, ISCA '07.

[18]  Charles Yount,et al.  Using Model Trees for Computer Architecture Performance Analysis of Software Applications , 2007, 2007 IEEE International Symposium on Performance Analysis of Systems & Software.

[19]  Darryl Gove,et al.  CPU2006 working set size , 2007, CARN.

[20]  Benjamin C. Lee,et al.  Statistically Rigorous Regression Modeling for the Microprocessor Design Space , 2006 .

[21]  John L. Henning.  Performance counters and development of SPEC CPU2006 , 2007, CARN.

[22]  Lizy Kurian John,et al.  Subsetting the SPEC CPU2006 benchmark suite , 2007, CARN.