Finding representative workloads for computer system design

This work explores how improved workload characterization can be used for a better selection of representative workloads within the computer system and processor design process. We find that metrics easily available in modern computer systems provide sufficient distinctive capability for workload selection, thus avoiding the need to characterize a large number of workloads in simulation.

[1]  Marina MeWi Comparing Clusterings , 2002 .

[2]  Milo M. K. Martin,et al.  Simulating a $ 2 M Commercial Server on a $ 2 K PC T , 2001 .

[3]  Matteo Golfarelli,et al.  The workload you have, the workload you would like , 2003, DOLAP '03.

[4]  Francisco Tirado,et al.  Analysis of simulation-adapted SPEC 2000 benchmarks , 2002, CARN.

[5]  Fredrik Larsson,et al.  Simics: A Full System Simulation Platform , 2002, Computer.

[6]  Todd M. Austin,et al.  SimpleScalar: An Infrastructure for Computer System Modeling , 2002, Computer.

[7]  Mark Crovella,et al.  Computer Systems Performance Evaluation , 2007 .

[8]  J. Dujmovi State-of-the-art in Performance Modeling and Simulation: Theory, Techniques, and Tutorials. Edited Evaluation and Design of Benchmark Suites , 1996 .

[9]  Jozo J. Dujmovic Universal benchmark suites , 1999, MASCOTS '99. Proceedings of the Seventh International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems.

[10]  W. Cleveland Robust Locally Weighted Regression and Smoothing Scatterplots , 1979 .

[11]  Giuseppe Serazzi,et al.  Workload characterization: a survey , 1993, Proc. IEEE.

[12]  Koen De Bosschere,et al.  Many benchmarks stress the same bottlenecks , 2004 .

[13]  Paul S. Bradley,et al.  Refining Initial Points for K-Means Clustering , 1998, ICML.

[14]  J. M. Mohr,et al.  A model for workload characterization , 1975, ANSS '75.

[15]  T. Kuhn,et al.  The Structure of Scientific Revolutions. , 1964 .

[16]  Qing Wang,et al.  Workload characterization for an E-commerce web site , 2003, CASCON.

[17]  Carey L. Williamson,et al.  A workload characterization study of Internet Web servers , 1995, SIAP.

[18]  Alan Jay Smith,et al.  Machine Characterization Based on an Abstract High-Level Language Machine , 1989, IEEE Trans. Computers.

[19]  David J. Sheskin,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[20]  M. Meilă Comparing clusterings---an information based distance , 2007 .

[21]  Lieven Eeckhout,et al.  Measuring Program Similarity , 2005 .

[22]  Alan Jay Smith,et al.  Analysis of benchmark characteristics and benchmark performance prediction , 1996, TOCS.

[23]  Harry F. Jordan,et al.  Computer systems design and architecture , 1996 .

[24]  Anand Sivasubramaniam,et al.  Characterizing the d-TLB behavior of SPEC CPU2000 benchmarks , 2002, SIGMETRICS '02.

[25]  Ali S. Hadi,et al.  Finding Groups in Data: An Introduction to Chster Analysis , 1991 .

[26]  Mikko H. Lipasti,et al.  A performance methodology for commercial servers , 2000, IBM J. Res. Dev..

[27]  Adrian E. Raftery,et al.  MCLUST: Software for Model-Based Clustering, Density Estimation and Discriminant Analysis , 2002 .

[28]  Lodewijk Bonebakker,et al.  Quantifying hardware counter sampling error in computer system workload characterization , 2007 .

[29]  Joseph Pasquale,et al.  A static and dynamic workload characterization study of the San Diego Supercomputer center Cray X-MP , 1991, SIGMETRICS '91.

[30]  Marina Meila,et al.  Comparing Clusterings by the Variation of Information , 2003, COLT.

[31]  Gary Lauterbach,et al.  UltraSPARC-III: designing third-generation 64-bit performance , 1999, IEEE Micro.

[32]  Lieven Eeckhout,et al.  Performance prediction based on inherent program similarity , 2006, 2006 International Conference on Parallel Architectures and Compilation Techniques (PACT).

[33]  Roland N. Ibbett,et al.  Technical note: a hierarchical computer architecture design and simulation environment , 1998, TOMC.

[34]  Pradip Bose,et al.  Performance Analysis and Its Impact on Design , 1998, Computer.

[35]  Lizy K. John,et al.  Workload characterization: motivation, goals and methodology , 1998, Workload Characterization: Methodology and Case Studies. Based on the First Workshop on Workload Characterization.

[36]  Thin-Fong Tsuei,et al.  Queuing Simulation Model for Multiprocessor Systems , 2003, Computer.

[37]  T. Puzak,et al.  The optimum pipeline depth for a microprocessor , 2002, Proceedings 29th Annual International Symposium on Computer Architecture.

[38]  Wei-Chien Chang On using Principal Components before Separating a Mixture of Two Multivariate Normal Distributions , 1983 .

[39]  G. Celeux,et al.  A Classification EM algorithm for clustering and two stochastic versions , 1992 .

[40]  D.J. Lilja,et al.  Accurate statistical approaches for generating representative workload compositions , 2005, IEEE International. 2005 Proceedings of the IEEE Workload Characterization Symposium, 2005..

[41]  Jr. Frederick P. Brooks,et al.  The mythical man-month (anniversary ed.) , 1995 .

[42]  Daniel D. Lee,et al.  Multiplicative Updates for Nonnegative Quadratic Programming in Support Vector Machines , 2002, NIPS.

[43]  B. Ramakrishna Rau,et al.  EPIC: An Architecture for Instruction-Level Parallel Processors , 2000 .

[44]  J. MacQueen Some methods for classification and analysis of multivariate observations , 1967 .

[45]  Mendel Rosenblum,et al.  The Reincarnation of Virtual Machines , 2004, ACM Queue.

[46]  John A. Hartigan,et al.  Clustering Algorithms , 1975 .

[47]  A. Raftery,et al.  Model-based Gaussian and non-Gaussian clustering , 1993 .

[48]  Jeffrey S. Vetter,et al.  Scalable Analysis Techniques for Microprocessor Performance Counter Metrics , 2002, ACM/IEEE SC 2002 Conference (SC'02).

[49]  Jozo J. Dujmovic,et al.  Evolution and evaluation of SPEC benchmarks , 1998, PERV.

[50]  Koen De Bosschere,et al.  Eccentric and fragile benchmarks , 2004, IEEE International Symposium on - ISPASS Performance Analysis of Systems and Software, 2004.

[51]  Kunle Olukotun,et al.  The case for a single-chip multiprocessor , 1996, ASPLOS VII.

[52]  Lieven Eeckhout,et al.  Quantifying the Impact of Input Data Sets on Program Behavior and its Applications , 2003, J. Instr. Level Parallelism.

[53]  Neil J. Gunther,et al.  The Practical Performance Analyst , 1998 .

[54]  James E. Smith,et al.  The future of simulation: a field of dreams , 2006, Computer.

[55]  A. P. Sage Systems engineering of computer based systems: status and future perspectives , 1995, Proceedings of the 1995 International Symposium and Workshop on Systems Engineering of Computer-Based Systems.

[56]  J. A. Hartigan,et al.  A k-means clustering algorithm , 1979 .

[57]  Eric Sprangle,et al.  Increasing processor performance by implementing deeper pipelines , 2002, ISCA.

[58]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[59]  Harsh Sharangpani,et al.  Itanium Processor Microarchitecture , 2000, IEEE Micro.

[60]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[61]  H. Vandierendonck,et al.  Experiments with subsetting benchmark suites , 2004, IEEE International Workshop on Workload Characterization, 2004. WWC-7. 2004.

[62]  Margaret Martonosi,et al.  Challenges in Computer Architecture Evaluation , 2003, Computer.

[63]  Kingsum Chow,et al.  Characterization of Java/sup TM/ application server workloads , 2001 .

[64]  Pradip Bose,et al.  Guest Editors' Introduction: Challenges in Processor Modeling and Validation , 1999 .

[65]  Domenico Ferrari,et al.  A sensitivity study of the clustering approach to workload modeling (extended abstract) , 1985, SIGMETRICS 1985.

[66]  Christopher Connelly,et al.  A workload characterization for coarse-grain multiprocessors , 1995, Proceedings of 9th International Parallel Processing Symposium.

[67]  John Leaney,et al.  Development of a systems architecting process for computer based systems , 1996, Proceedings of ICECCS '96: 2nd IEEE International Conference on Engineering of Complex Computer Systems (held jointly with 6th CSESAW and 4th IEEE RTAW).

[68]  Kunle Olukotun,et al.  Niagara: a 32-way multithreaded Sparc processor , 2005, IEEE Micro.

[69]  Adrian E. Raftery,et al.  Model-Based Clustering, Discriminant Analysis, and Density Estimation , 2002 .

[70]  H. R. Simpson,et al.  Architecture for computer based systems , 1994, Proceedings 1994 Tutorial and Workshop on Systems Engineering of Computer-Based Systems.

[71]  J. Ramsay Monotone Regression Splines in Action , 1988 .

[72]  Erkki Oja,et al.  Independent component analysis: algorithms and applications , 2000, Neural Networks.

[73]  Richard McDougall,et al.  Solaris Internals: Solaris 10 and OpenSolaris Kernel Architecture , 2006 .

[74]  David A. Patterson,et al.  Truth in SPEC benchmarks , 1995, CARN.

[75]  Ashok K. Agrawala,et al.  An Approach to the Workload Characterization Problem , 1976, Computer.

[76]  Lothar Thiele,et al.  Workload characterization model for tasks with variable execution demand , 2004, Proceedings Design, Automation and Test in Europe Conference and Exhibition.

[77]  Thomas M. Cover,et al.  Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing) , 2006 .

[78]  Lieven Eeckhout,et al.  Workload design: selecting representative program-input pairs , 2002, Proceedings.International Conference on Parallel Architectures and Compilation Techniques.

[79]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[80]  K. M. Khalil,et al.  LAN traffic analysis and workload characterization , 1990, [1990] Proceedings. 15th Conference on Local Computer Networks.

[81]  Philip S. Yu,et al.  On Workload Characterization of Relational Database Environments , 1992, IEEE Trans. Software Eng..

[82]  Fang Liu,et al.  Accelerating Full-System Simulation through Characterizing and Predicting Operating System Performance , 2007, 2007 IEEE International Symposium on Performance Analysis of Systems & Software.

[83]  Erkki Oja,et al.  Independent Component Analysis , 2001 .

[84]  Adrian E. Raftery,et al.  How Many Clusters? Which Clustering Method? Answers Via Model-Based Cluster Analysis , 1998, Comput. J..

[85]  Martin Arlitt,et al.  A workload characterization study of the 1998 World Cup Web site , 2000, IEEE Netw..

[86]  Daniel A. Menascé Workload Characterization , 2003, IEEE Internet Comput..

[87]  Benjamin C. Lee An Architectural Assessment of SPEC CPU Benchmark Relevance , 2006 .

[88]  Aapo Hyvärinen,et al.  Survey on Independent Component Analysis , 1999 .

[89]  Rohit Bhatia,et al.  Montecito: a dual-core, dual-thread Itanium processor , 2005, IEEE Micro.

[90]  Neil J. Gunther,et al.  The Practical Performance Analyst: Performance-by-Design Techniques for Distributed Systems , 1997 .

[91]  Salvatore T. March,et al.  Design and natural science research on information technology , 1995, Decis. Support Syst..

[92]  David A. Patterson,et al.  Computer Architecture: A Quantitative Approach , 1969 .

[93]  Lieven Eeckhout,et al.  Measuring benchmark similarity using inherent program characteristics , 2006, IEEE Transactions on Computers.

[94]  Adrian E. Raftery,et al.  Enhanced Model-Based Clustering, Density Estimation, and Discriminant Analysis Software: MCLUST , 2003, J. Classif..

[95]  Michael W. Berry,et al.  Scientific workload characterization by loop-based analyses , 1992, PERV.

[96]  Douglas M. Hawkins,et al.  A statistically rigorous approach for improving simulation methodology , 2003, The Ninth International Symposium on High-Performance Computer Architecture, 2003. HPCA-9 2003. Proceedings..

[97]  Matthew Chapman,et al.  Itanium: a system implementor's tale , 2005 .

[98]  Paul H. J. Kelly,et al.  Workload characterization using lightweight system call tracing and reexecution , 1998, 1998 IEEE International Performance, Computing and Communications Conference. Proceedings (Cat. No.98CH36191).

[99]  Giuseppe Serazzi,et al.  Measurement and Tuning of Computer Systems , 1984, Int. CMG Conference.

[100]  D. Patterson,et al.  Performance characterization of a quad Pentium Pro SMP using OLTP workloads , 1998, Proceedings. 25th Annual International Symposium on Computer Architecture (Cat. No.98CB36235).

[101]  D. Citron MisSPECulation: partial and misleading use of spec CPU2000 in computer architecture conferences , 2003, 30th Annual International Symposium on Computer Architecture, 2003. Proceedings..

[102]  Raj Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.

[103]  Quinn Snell,et al.  HINT: A new way to measure computer performance , 1995, Proceedings of the Twenty-Eighth Annual Hawaii International Conference on System Sciences.

[104]  Tsunenori Ishioka,et al.  Extended K-means with an Efficient Estimation of the Number of Clusters , 2000, Ideal.

[105]  Domenico Ferrari,et al.  Workload charaterization and Selection in Computer Performance Measurement , 1972, Computer.

[106]  I. Ahmad,et al.  An analysis of disk performance in VMware ESX server virtual machines , 2003, 2003 IEEE International Conference on Communications (Cat. No.03CH37441).

[107]  Norman P. Jouppi,et al.  Enterprise IT trends and implications for architecture research , 2005, 11th International Symposium on High-Performance Computer Architecture.

[108]  Trevor N. Mudge,et al.  Systematic objective-driven computer architecture optimization , 1995, Proceedings Sixteenth Conference on Advanced Research in VLSI.

[109]  Pradeep K. Dubey,et al.  How Multimedia Workloads Will Change Processor Design , 1997, Computer.

[110]  Martin F. Arlitt,et al.  Web server workload characterization: the search for invariants , 1996, SIGMETRICS '96.

[111]  Brian Randell,et al.  Iterative multi-level modelling. A methodology for computer system design , 1968, IFIP Congress.

[112]  Tom W. Keller,et al.  A workload characterization pipeline for models of parallel systems , 1987, SIGMETRICS '87.

[113]  Lizy Kurian John,et al.  Workload characterization of multithreaded java servers , 2001, 2001 IEEE International Symposium on Performance Analysis of Systems and Software. ISPASS..

[114]  Adrian E. Raftery,et al.  MCLUST: Software for Model-Based Cluster Analysis , 1999 .

[115]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[116]  Larry F. Hodges,et al.  Workload characterization and performance evaluation in a research environment , 1982, SIGMETRICS '82.

[117]  David Kotz,et al.  Dynamic file-access characteristics of a production parallel scientific workload , 1994, Proceedings of Supercomputing '94.

[118]  Christoph Schaffer Hierarchical architectural design, simulation and evaluation , 1996, Proceedings IEEE Symposium and Workshop on Engineering of Computer-Based Systems.

[119]  Victor R. Basili,et al.  Iterative and incremental developments. a brief history , 2003, Computer.

[120]  Ayman I. Kayssi,et al.  The design of a microsupercomputer , 1991, Computer.

[121]  Steven R. Kunkel,et al.  A multithreaded PowerPC processor for commercial servers , 2000, IBM J. Res. Dev..

[122]  E. Tufte Beautiful Evidence , 2006 .

[123]  Edward S. Davidson,et al.  Computer system design using a hierarchical approach to performance evaluation , 1980, CACM.

[124]  R. Tibshirani,et al.  Generalized Additive Models , 1991 .

[125]  Lieven Eeckhout,et al.  Measuring Program Similarity: Experiments with SPEC CPU Benchmark Suites , 2005, IEEE International Symposium on Performance Analysis of Systems and Software, 2005. ISPASS 2005..

[126]  Mark S. Squillante,et al.  Models of Parallel Applications with Large Computation and I/O Requirements , 2002, IEEE Trans. Software Eng..

[127]  Mariacarla Calzarossa,et al.  Workload Characterization Issues and Methodologies , 2000, Performance Evaluation.

[128]  K. Olukotun,et al.  Evaluation of Design Alternatives for a Multiprocessor Microprocessor , 1996, 23rd Annual International Symposium on Computer Architecture (ISCA'96).

[129]  Santosh G. Abraham,et al.  Chip multithreading: opportunities and challenges , 2005, 11th International Symposium on High-Performance Computer Architecture.

[130]  David A. Patterson,et al.  Computer architecture (2nd ed.): a quantitative approach , 1996 .

[131]  Barry W. Boehm,et al.  A spiral model of software development and enhancement , 1986, Computer.

[132]  Douglas M. Hawkins,et al.  Improving computer architecture simulation methodology by adding statistical rigor , 2005, IEEE Transactions on Computers.

[133]  Andrew W. Moore,et al.  X-means: Extending K-means with Efficient Estimation of the Number of Clusters , 2000, ICML.

[134]  Kimmo E. E. Raatikainen,et al.  Cluster analysis and workload classification , 1993, PERV.

[135]  John Flynn,et al.  Adapting the SPEC 2000 benchmark suite for simulation-based computer architecture research , 2001 .

[136]  Carey L. Williamson,et al.  Internet Web servers: workload characterization and performance implications , 1997, TNET.

[137]  Lieven Eeckhout,et al.  FOUR GENERATIONS OF SPEC CPU BENCHMARKS : WHAT HAS CHANGED AND WHAT HAS NOT , 2004 .

[138]  E. M. Wright,et al.  Adaptive Control Processes: A Guided Tour , 1961, The Mathematical Gazette.

[139]  Ethem Alpaydin,et al.  Introduction to Machine Learning (Adaptive Computation and Machine Learning) , 2004 .

[140]  Lizy K. John,et al.  Workload Characterization of Emerging Computer Applications , 2001 .

[141]  Tsunenori Ishioka,et al.  An Expansion of X-Means for Automatically Determining the Optimal Number of Clusters a^EUR" Progressive Iterations of K-Means and Merging of the Clusters , 2005, Computational Intelligence.

[142]  Antonino Mazzeo,et al.  Workload characterization for trend analysis , 1981, PERV.

[143]  L. Eeckhout,et al.  Exploiting program microarchitecture independent characteristics and phase behavior for reduced benchmark suite simulation , 2005, IEEE International. 2005 Proceedings of the IEEE Workload Characterization Symposium, 2005..

[144]  Sandhya Dwarkadas,et al.  Characterizing and predicting program behavior and its variability , 2003, 2003 12th International Conference on Parallel Architectures and Compilation Techniques.

[145]  W. Cleveland LOWESS: A Program for Smoothing Scatterplots by Robust Locally Weighted Regression , 1981 .

[146]  Lieven Eeckhout,et al.  Designing Computer Architecture Research Workloads , 2003, Computer.

[147]  Lieven Eeckhout,et al.  Comparing Benchmarks Using Key Microarchitecture-Independent Characteristics , 2006, 2006 IEEE International Symposium on Workload Characterization.

[148]  Frank F. Land,et al.  Viewpoint: choosing appropriate information systems research methodologies , 1987, CACM.

[149]  Michael J. Flynn,et al.  Computer Architecture: Pipelined and Parallel Processor Design , 1995 .

[150]  Louise Trevillyan,et al.  EDA in IBM: past, present, and future , 2000, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[151]  David J. Lilja,et al.  Simulation of computer architectures: simulators, benchmarks, methodologies, and recommendations , 2006, IEEE Transactions on Computers.

[152]  Lieven Eeckhout,et al.  Efficient microprocessor design space exploration through statistical simulation , 2003, 36th Annual Simulation Symposium, 2003..