Performance Prediction and Ranking of Supercomputers

Abstract Performance prediction indicates the time required for execution of an application on a particular machine. Machine ranking indicates the set of machines that is likely to execute an application most quickly. These two questions are discussed within the context of large parallel applications run on on supercomputers. Different techniques are surveyed, including a framework for a general approach that weighs the results of machine benchmarks run on all systems of interest. Variations within the framework are described and tested on data from large-scale applications run on modern supercomputers, helping to illustrate the trade-offs in accuracy and effort that are inherent in any method for answering these two questions.

[1]  Michael Laurenzano,et al.  How well can simple metrics represent the performance of HPC applications? , 2005, ACM/IEEE SC 2005 Conference (SC'05).

[2]  F. Berman,et al.  Adaptive Performance Prediction for Distributed Data-Intensive Applications , 1999, ACM/IEEE SC 1999 Conference (SC'99).

[3]  Frank Mueller,et al.  Cross-Platform Performance Prediction of Parallel Applications Using Partial Execution , 2005, ACM/IEEE SC 2005 Conference (SC'05).

[4]  Jesús Labarta,et al.  A Framework for Performance Modeling and Prediction , 2002, ACM/IEEE SC 2002 Conference (SC'02).

[5]  Jesús Labarta,et al.  Deriving analytical models from a limited number of runs , 2003, PARCO.

[6]  Eike Jessen,et al.  Workshop on Wide Area Networks and High Performance Computing , 1998 .

[7]  Vikram S. Adve,et al.  Analyzing the behavior and performance of parallel programs , 1993 .

[8]  Adolfy Hoisie,et al.  A performance comparison between the Earth Simulator and other terascale systems on a characteristic ASCI workload , 2005, Concurr. Pract. Exp..

[9]  Michael J. Flynn,et al.  Detection and Parallel Execution of Independent Instructions , 1970, IEEE Transactions on Computers.

[10]  Daniel A. Reed,et al.  Integrated compilation and scalability analysis for parallel systems , 1998, Proceedings. 1998 International Conference on Parallel Architectures and Compilation Techniques (Cat. No.98EX192).

[11]  Thomas J. LeBlanc,et al.  Parallel performance prediction using lost cycles analysis , 1994, Proceedings of Supercomputing '94.

[12]  L. J. Boland,et al.  The IBM system/360 model 91: storage system , 1967 .

[13]  Jack Dongarra,et al.  Introduction to the HPCChallenge Benchmark Suite , 2004 .

[14]  Daniel P. Spooner,et al.  Identification of Performance Characteristics from Multi-view Trace Analysis , 2003, International Conference on Computational Science.

[15]  Alan Jay Smith,et al.  Measuring Cache and TLB Performance and Their Effect on Benchmark Runtimes , 1995, IEEE Trans. Computers.

[16]  Doug Burger,et al.  Evaluating Future Microprocessors: the SimpleScalar Tool Set , 1996 .

[17]  John M. Mellor-Crummey,et al.  Cross-architecture performance predictions for scientific applications using parameterized models , 2004, SIGMETRICS '04/Performance '04.

[18]  Lin Sun,et al.  Semi-Empirical Multiprocessor Performance Predictions , 1996, J. Parallel Distributed Comput..

[19]  Hans Werner Meuer,et al.  Top500 Supercomputer Sites , 1997 .

[20]  Jack J. Dongarra,et al.  The LINPACK Benchmark: past, present and future , 2003, Concurr. Comput. Pract. Exp..

[21]  Laura Carrington,et al.  A performance prediction framework for scientific applications , 2003, Future Gener. Comput. Syst..

[22]  Adolfy Hoisie,et al.  Performance Analysis of Wavefront Algorithms on Very-Large Scale Distributed Systems , 1998, Wide Area Networks and High Performance Computing.

[23]  Laura Carrington,et al.  A Framework for Application Performance Modeling and Prediction , 2002 .

[24]  Mark J. Clement,et al.  Multivariate statistical techniques for parallel performance prediction , 1995, Proceedings of the Twenty-Eighth Annual Hawaii International Conference on System Sciences.

[25]  Sally A. McKee,et al.  An Approach to Performance Prediction for Parallel Applications , 2005, Euro-Par.

[26]  PredictionCelso L. Mendes,et al.  Performance Stability and Prediction , 1994 .

[27]  Alan Jay Smith,et al.  Performance Characterization of Optimizing Compilers , 1992, IEEE Trans. Software Eng..

[28]  William T. C. Kramer,et al.  Performance Variability of Highly Parallel Architectures , 2003, International Conference on Computational Science.

[29]  Ramesh Subramonian,et al.  LogP: towards a realistic model of parallel computation , 1993, PPOPP '93.

[30]  Yong Luo,et al.  Development and validation of a hierarchical memory model incorporating CPU- and memory-operation overlap model , 1998, WOSP '98.

[31]  Adolfy Hoisie,et al.  Exploring advanced architectures using performance prediction , 2002, International Workshop on Innovative Architecture for Future Generation High-Performance Processors and Systems.

[32]  Omid Khalili,et al.  Performance Prediction and Ordering of Supercomputers using a Linear Combination of Benchmark Measurements , 2007 .

[33]  Dean M. Tullsen,et al.  Converting thread-level parallelism to instruction-level parallelism via simultaneous multithreading , 1997, TOCS.

[34]  F. Petrini,et al.  The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[35]  Sally A. McKee,et al.  Predicting parallel application performance via machine learning approaches , 2007, Concurr. Comput. Pract. Exp..

[36]  Darren J. Kerbyson,et al.  A Performance Model of the Parallel Ocean Program , 2005, Int. J. High Perform. Comput. Appl..

[37]  Alan Jay Smith,et al.  Analysis of benchmark characteristics and benchmark performance prediction , 1996, TOCS.

[38]  Sally A. McKee,et al.  Hitting the memory wall: implications of the obvious , 1995, CARN.

[39]  Jens Simon,et al.  Accurate Performance Prediction for Assively Parallel Systems and Its Applications , 1996, Euro-Par, Vol. II.

[40]  Adolfy Hoisie,et al.  Scalability analysis of multidimensional wavefront algorithms on large-scale SMP clusters , 1999, Proceedings. Frontiers '99. Seventh Symposium on the Frontiers of Massively Parallel Computation.