Comparison of the SPEC CPU Benchmarks with 499 Other Workloads Using Hardware Counters

This work extends an existing workload comparison approach used for simulation based metrics to computer system based metrics. We apply this approach using processor hardware counters and compare characterizations of spec cpu2000 and spec cpu2006, against real (commercial) workloads and other benchmarks collected on the same computer system architecture. Using Independent Component analysis we reduce the many dimensional workload characterization space into a lesser dimensional representative space prior to comparing the distribution of workloads. We find that the spec cpu benchmarks are for the most part representative on the identified principal components, with notable exceptions.

[1]  Lieven Eeckhout,et al.  Measuring benchmark similarity using inherent program characteristics , 2006, IEEE Transactions on Computers.

[2]  Ann Marie Grizzaffi Maynard,et al.  Contrasting characteristics and cache performance of technical and multi-user commercial workloads , 1994, ASPLOS VI.

[3]  Erkki Oja,et al.  Independent component analysis: algorithms and applications , 2000, Neural Networks.

[4]  Kimmo E. E. Raatikainen,et al.  Cluster analysis and workload classification , 1993, PERV.

[5]  Lodewijk Bonebakker,et al.  Quantifying hardware counter sampling error in computer system workload characterization , 2007 .

[6]  I. Jolliffe.  Principal Component Analysis , 2002 .

[7]  Daniel Citron.  MisSPECulation: partial and misleading use of SPEC CPU2000 in computer architecture conferences , 2003, ISCA '03.

[8]  Lieven Eeckhout,et al.  Measuring Program Similarity: Experiments with SPEC CPU Benchmark Suites , 2005, IEEE International Symposium on Performance Analysis of Systems and Software, 2005. ISPASS 2005..

[9]  Pradeep K. Dubey,et al.  How Multimedia Workloads Will Change Processor Design , 1997, Computer.

[10]  Weifeng Liu,et al.  Adaptive and Learning Systems for Signal Processing, Communication, and Control , 2010 .

[11]  Jan Lodewijk Bonebakker,et al.  Finding representative workloads for computer system design , 2007 .

[12]  Koen De Bosschere,et al.  Eccentric and fragile benchmarks , 2004, IEEE International Symposium on - ISPASS Performance Analysis of Systems and Software, 2004.

[13]  Stamatis Vassiliadis,et al.  GraalBench: a 3D graphics benchmark suite for mobile phones , 2004, LCTES '04.

[14]  D. B. Davis,et al.  Sun Microsystems Inc. , 1993 .

[15]  Lieven Eeckhout,et al.  Workload design: selecting representative program-input pairs , 2002, Proceedings.International Conference on Parallel Architectures and Compilation Techniques.

[16]  E. Oja,et al.  Independent Component Analysis , 2013 .

[17]  David J. Sheskin,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[18]  H. Vandierendonck,et al.  Experiments with subsetting benchmark suites , 2004, IEEE International Workshop on Workload Characterization, 2004. WWC-7. 2004.

[19]  D.J. Lilja,et al.  Accurate statistical approaches for generating representative workload compositions , 2005, IEEE International. 2005 Proceedings of the IEEE Workload Characterization Symposium, 2005..