Computer Aided Performance Engineering: Current State of the Art. Addendum to Final Technical Report,

Abstract : This is an addendum to the Final Technical Report submitted as a result of Phase I SBIR work awarded to Infopike, Inc. This addendum reflects the work done by us in assessing the state of the art in the field of Computer Aided Performance Engineering. Performance evaluation of computer and software systems has become a rapidly growing field with a growing number of tools being developed for analyzing various performance aspects of such systems. The literature on performance evaluation methodologies has also mushroomed with various proposals from researchers all over the world. This addendum presents the results of a survey conducted on automated performance analysis tools for computer systems. The survey includes measurement-based tools, analytical tools, simulation tools and visualization tools, and describes their properties and capabilities. The tools have been categorized based on their analysis capabilities and include system-oriented, process-oriented and module-oriented categories. The tools surveyed in this paper incorporate various techniques including simulation, modeling (Petri net, queuing, semi-markov etc.), measurement, visualization and emulation. We also present a new methodology for tool classification that aids system designers.

[1]  Franz Sötz,et al.  A Method for Performance Prediction of Parallel Programs , 1990, CONPAR.

[2]  Connie U. Smith,et al.  SPE Evaluation of a Client/Server Application , 1994, Int. CMG Conference.

[3]  Udo R. Krieger,et al.  Modeling and Analysis of Communication Systems Based on Computational Methods for Markov Chains , 1990, IEEE J. Sel. Areas Commun..

[4]  Taylor L. Booth,et al.  Software optimization using user models , 1988, IEEE Trans. Syst. Man Cybern..

[5]  R. Blasko.  Hierarchical performance prediction for parallel programs , 1995, Proceedings of the 1995 International Symposium and Workshop on Systems Engineering of Computer-Based Systems.

[6]  Thomas Fahringer,et al.  A static parameter based performance prediction tool for parallel programs , 1993, ICS '93.

[7]  Robert J. T. Morris,et al.  Visual Simulation: The Performance Analysis Workstation , 1985, Computer.

[8]  Matthias Schumann.  Efficient performance prediction for parallel programs , 1996 .

[9]  Paul A. Fishwick,et al.  SimPack: getting started with simulation programming in C and C++ , 1992, WSC '92.

[10]  John Stasko,et al.  The PARADE Environment for Visualizing Parallel Program Executions: A Progress Report , 1995 .

[11]  Edward D. Lazowska,et al.  Quantitative System Performance , 1985, Int. CMG Conference.

[12]  Scott A. Smolka,et al.  Integrated Environments for Formally Well-Founded Design and Simulation of Concurrent Systems , 1988, IEEE Trans. Software Eng..

[13]  Boris Beizer.  Micro-analysis of computer system performance , 1978 .

[14]  Alexander V. Veidenbaum,et al.  Chief: a parallel simulation environment for parallel systems , 1991, [1991] Proceedings. The Fifth International Parallel Processing Symposium.

[15]  Ken Kennedy,et al.  A static performance estimator to guide data partitioning decisions , 1991, PPOPP '91.

[16]  Jerry C. Yan,et al.  “AXE”: a simulation environment for actor-like computations on ensemble architectures , 1986, WSC '86.

[17]  Anant Agarwal,et al.  Multiprocessor cache analysis using ATUM , 1988, ISCA '88.

[18]  P. A. Bobillier,et al.  Simulation With Gpss and Gpss V , 1976 .

[19]  Christoph Lindemann,et al.  DSPNexpress: A Software Package for the Efficient Solution of Deterministic and Stochastic Petri Nets , 1993, Perform. Evaluation.

[20]  Edward D. Lazowska,et al.  Speedup Versus Efficiency in Parallel Systems , 1989, IEEE Trans. Computers.

[21]  J. P. Prost,et al.  Modeling Methodology for Performance Evaluation of Parallel Architectures a Case Study, LCAP , 1989, Int. J. High Speed Comput..

[22]  Alan Sussman.  Model-driven mapping of computation onto distributed memory parallel computers , 1992 .

[23]  鈴木 敬,et al.  Efficient Software Performance Estimation Methods for Hardware/Software Codesign , 1996 .

[24]  Jerome A. Rolia,et al.  Automatic generation of a software performance model using an object-oriented prototype , 1995, MASCOTS '95. Proceedings of the Third International Workshop on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems.

[25]  Erol Gelenbe.  Performance analysis of the connection machine , 1990, SIGMETRICS '90.

[26]  Robert J. Fowler,et al.  MINT: a front end for efficient simulation of shared-memory multiprocessors , 1994, Proceedings of International Workshop on Modeling, Analysis and Simulation of Computer and Telecommunication Systems.

[27]  John L. Hennessy,et al.  Performance debugging shared memory multiprocessor programs with MTOOL , 1991, Proceedings of the 1991 ACM/IEEE Conference on Supercomputing (Supercomputing '91).

[28]  R.J. Block,et al.  Automated Performance Prediction of Message-Passing Parallel Programs , 1995, Proceedings of the IEEE/ACM SC95 Conference.

[29]  Jerry C. Yan,et al.  Performance Evaluation Tools for Parallel and Distributed Systems - Guest Editors' Introduction , 1995, Computer.

[30]  Xiaodong Zhang,et al.  Performance Prediction and Evaluation of Parallel Processing on a NUMA Multiprocessor , 1991, IEEE Trans. Software Eng..

[31]  Brian A. Malloy,et al.  SIMCAL: the merger of Simula and Pascal , 1986, WSC '86.

[32]  Friedemann Mattern,et al.  Key Concepts of the INCAS Multicomputer Project , 1987, IEEE Transactions on Software Engineering.

[33]  Rob Pooley.  Performance Analysis Tools in Europe , 1995, Informationstechnik Tech. Inform..

[34]  Karl-Heinz John,et al.  Experiences with Performance Measurement and Modeling of a Processor Array , 1983, IEEE Transactions on Computers.

[35]  R. D. Trammell.  The Big Picture: Visualizing System Behavior in Real Time , 1990, USENIX Summer.

[36]  Alan Mink,et al.  Hardware-Assisted Multiprocessor Performance Measurement , 1987, Performance.

[37]  Paul Rovner,et al.  User program measurement in a time-shared environment , 1971, CACM.

[38]  James B. Sinclair,et al.  Computer performance evaluation with GIST: a tool for specifying extended queueing network models , 1985, WSC '85.

[39]  Eric Foxley,et al.  Monitoring the run‐time activity of algol 68‐R programs , 1978, Softw. Pract. Exp..

[40]  Kathleen M. Nichols,et al.  Modeling multicomputer systems with PARET , 1988, Computer.

[41]  Allen D. Malony,et al.  Performance prediction of loop constructs on multiprocessor hierarchical-memory systems , 1989, ICS '89.

[42]  S. Madala,et al.  Performance of Synchronous Parallel Algorithms with Regular Structures , 1991, IEEE Trans. Parallel Distributed Syst..

[43]  Barton P. Miller,et al.  IPS-2: The Second Generation of a Parallel Program Measurement System , 1990, IEEE Trans. Parallel Distributed Syst..

[44]  Jacques Cohen,et al.  Computer-assisted microanalysis of parallel programs , 1992, TOPL.

[45]  Robert J Carpenter.  ICST-Performance Measurement Instrumentation for Multiprocessor Computers , 1987 .

[46]  Robert M. O'Keefe,et al.  Discrete visual simulation with Pascal_SIM , 1986, WSC '86.

[47]  Carlos Juiz,et al.  Approximate performance models of real-time software systems , 1995, MASCOTS '95. Proceedings of the Third International Workshop on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems.

[48]  Lester Lipsky,et al.  An architecture assessment environment for massively parallel computations , 1989, Conference Proceedings., IEEE International Conference on Systems, Man and Cybernetics.

[49]  John P. Penny,et al.  Data Recording and Monitoring for Analysis of System Response Times , 1986, Comput. J..

[50]  Jacques Cohen,et al.  Software tools for micro‐analysis of programs , 1992, Softw. Pract. Exp..

[51]  Ken Kennedy,et al.  An Integrated Compilation and Performance Analysis Environment for Data Parallel Programs , 1995, SC.

[52]  Alan H. Karp,et al.  Measuring parallel processor performance , 1990, CACM.

[53]  Ravishankar K. Iyer,et al.  DEPEND: a simulation environment for system dependability modeling and evaluation , 1996, Proceedings of IEEE International Computer Performance and Dependability Symposium.

[54]  Reda A. Ammar,et al.  A technique to derive the detailed time costs of parallel computations , 1988, Proceedings COMPSAC 88: The Twelfth Annual International Computer Software & Applications Conference.

[55]  Barton P. Miller,et al.  Performance Measurement for Parallel and Distributed Programs: A Structured and Automatic Approach , 1989, IEEE Trans. Software Eng..

[56]  Konrad Slind,et al.  Monitoring distributed systems , 1987, TOCS.

[57]  Reda A. Ammar,et al.  A design and modeling methodology for performance evaluation in real-time, distributed software systems , 1991, Conference Proceedings 1991 IEEE International Conference on Systems, Man, and Cybernetics.

[58]  Vivek Sarkar,et al.  On Estimating and Enhancing Cache Effectiveness , 1991, LCPC.

[59]  Mohamed Jamal Zemerly,et al.  A Layered Approach to the Characterisation of Parallel Systems for Performance Prediction , 1993 .

[60]  Ray Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.

[61]  Thomas L. Sterling,et al.  Multiprocessor Performance Measurement Using Embedded Instrumentation , 1988, ICPP.

[62]  Carl Kesselman.  Tools and techniques for performance measurement and performance improvement in parallel programs , 1991 .

[63]  D. Haban,et al.  Monitoring and performance measuring distributed systems during operation , 1988, SIGMETRICS 1988.

[64]  Allen D. Malony,et al.  Performance Prediction for Parallel Numerical Algorithms , 1991, Int. J. High Speed Comput..

[65]  Hermann Mierendorff,et al.  LAPAS: A Performance Evaluation Tool for Large Parallel Systems , 1990, ARCS.

[66]  David R. Hanson,et al.  Measuring the Performance and Behavior of Icon Programs , 1983, IEEE Transactions on Software Engineering.

[67]  Bin Qin.  Performance analysis of parallel computations , 1988 .

[68]  Chita R. Das,et al.  Performance Analysis of Cluster-Based Multiprocessors , 1994, IEEE Trans. Computers.

[69]  Marc Snir,et al.  The Performance of Multistage Interconnection Networks for Multiprocessors , 1983, IEEE Transactions on Computers.

[70]  Anoop Gupta,et al.  Complete computer system simulation: the SimOS approach , 1995, IEEE Parallel Distributed Technol. Syst. Appl..

[71]  Richard J. LeBlanc,et al.  Event-Driven Monitoring of Distributed Programs , 1985, ICDCS.

[72]  Edward A. MacNair.  An introduction to the Research Queueing Package , 1985, WSC '85.

[73]  Gordon Lyon,et al.  Simple transforms for instrumenting fortran decks , 1975, Softw. Pract. Exp..

[74]  Balkrishna Ramkumar,et al.  Performance prediction for portable parallel execution on MIMD architectures , 1995, Proceedings of 9th International Parallel Processing Symposium.

[75]  Thomas Fahringer.  Estimating and Optimizing Performance for Parallel Programs , 1995, Computer.

[76]  Norbert Götz,et al.  Multiprocessor and Distributed System Design: The Integration of Functional Specification and Performance Analysis Using Stochastic Process Algebras , 1993, Performance/SIGMETRICS Tutorials.

[77]  Claude C. Barnett.  Simulation in Pascal with micro PASSIM , 1986, WSC '86.

[78]  Giovanni Chiola,et al.  Stochastic Petri nets as a tools for the analysis of high-performance distributed architectures , 1989 .

[79]  K. Mani Chandy,et al.  A Message-Based Approach to Discrete-Event Simulation , 1987, IEEE Transactions on Software Engineering.

[80]  Ken Kennedy,et al.  Automatic Data Layout for Distributed-Memory Machines in the D Programming Environment , 1994, Automatic Parallelization.

[81]  Graham R. Nudd,et al.  A Layered Approach to Parallel Software Performance Prediction: A Case Study , 1994, EUROSIM.

[82]  Connie U. Smith,et al.  Performance analysis of software for an MIMD computer , 1982, SIGMETRICS '82.

[83]  Hans P. Zima,et al.  Automatic Support for Data Distribution , 1991, The Sixth Distributed Memory Computing Conference, 1991. Proceedings.

[84]  Kevin J. Healy.  CINEMA tutorial , 1985, Winter Simulation Conference.

[85]  Bernd Mohr,et al.  SIMPLE: A Performance Evaluation Tool Environment for Parallel and Distributed Systems , 1991, EDMCC.

[86]  Reda A. Ammar,et al.  Computer aided performance engineering: a survey , 1992 .

[87]  Paul F. Roth,et al.  Computer performance evaluation : tools and techniques for effective analysis , 1982 .

[88]  Barton P. Miller,et al.  A distributed programs monitor for berkeley UNIX , 1985, Softw. Pract. Exp..

[89]  Ben Wegbreit,et al.  Mechanical program analysis , 1975, CACM.

[90]  Hideyuki Tokuda,et al.  A real-time tool set for the ARTS kernel , 1988, Proceedings. Real-Time Systems Symposium.

[91]  Mark J. Clement,et al.  Performance comparison of desktop multiprocessing and workstation cluster computing , 1996, Proceedings of 5th IEEE International Symposium on High Performance Distributed Computing.

[92]  Dharma P. Agrawal,et al.  Performance of multiprocessor interconnection networks , 1989, Computer.

[93]  Anant Agarwal,et al.  Performance Tradeoffs in Multithreaded Processors , 1992, IEEE Trans. Parallel Distributed Syst..

[94]  Jean J. O'Reilly.  SLAM II: a tutorial , 1984, Winter Simulation Conference.

[95]  Reda A. Ammar,et al.  A tool to minimize the time costs of parallel computations through optimal processing power allocation , 1990, Softw. Pract. Exp..

[96]  J. Ramanujam,et al.  Compile-Time Techniques for Data Distribution in Distributed Memory Machines , 1991, IEEE Trans. Parallel Distributed Syst..

[97]  M. H. MacDougall.  Simulating computer systems: techniques and tools , 1989 .

[98]  Reda A. Ammar,et al.  A data modeling framework for the performance analysis of sequential and parallel software , 1993, CSC '93.

[99]  Verna Knapp,et al.  The Smalltalk simulation environment , 1986, WSC '86.

[100]  Charles H. Sauer,et al.  A Language for Extended Queuing Network Models , 1980, IBM J. Res. Dev..

[101]  Connie U. Smith,et al.  Performance Engineering of Software Systems , 1990, SIGMETRICS Perform. Evaluation Rev..

[102]  Hoon Choi,et al.  Performance Evaluation of Client-Server Systems , 1993, IEEE Trans. Parallel Distributed Syst..

[103]  Michael Gerndt,et al.  Work distribution in parallel programs for distributed memory multiprocessors , 1991, ICS '91.

[104]  D. Potier,et al.  QNAP 2:A portable environment for queueing systems modelling , 1984 .

[105]  Daniel T. Brunner,et al.  Advanced features of GPSS/H , 1987, WSC '87.

[106]  Ricardo F. Garzia.  Simulating with GPSS/PC , 1986, WSC '86.

[107]  C. D. Pegden,et al.  Introduction to SIMAN , 1990, 1990 Winter Simulation Conference Proceedings.

[108]  Herb Schwetman,et al.  Monit: a performance monitoring tool for parallel and pseudo-parallel programs , 1987, SIGMETRICS '87.

[109]  Thomas J. Schriber,et al.  Introduction to GPSS , 1976, WSC '79.

[110]  Reda A. Ammar,et al.  Visualizing a hierarchy of performance models for software systems , 1993, Softw. Pract. Exp..

[111]  Matthias Schumann.  Automatic performance prediction to support cross development of parallel programs , 1996, SPDT '96.

[112]  Roger W. Hockney.  A framework for benchmark performance analysis , 1991 .

[113]  M.J. Flynn,et al.  Analytical modeling of multithreaded pipeline performance , 1994, 1994 Proceedings of the Twenty-Seventh Hawaii International Conference on System Sciences.

[114]  Charles R. Standridge,et al.  A tutorial on TESS: the extended simulation system , 1984, Winter Simulation Conference.

[115]  Arif Ghafoor,et al.  PAWS: a performance evaluation tool for parallel computing systems , 1991, Computer.

[116]  James R. Spiegel,et al.  The flexible Ada simulation tool (FAST) and its extensions , 1987, WSC '87.

[117]  M.J. Flynn,et al.  Evaluating Performance Tradeoffs Between Fine-Grained and Coarse-Grained Alternatives , 1995, IEEE Trans. Parallel Distributed Syst..

[118]  Michael J. Quinn,et al.  A decomposition advisory system for heterogeneous data-parallel processing , 1994, Proceedings of 3rd IEEE International Symposium on High Performance Distributed Computing.

[119]  Mary K. Vernon,et al.  SARA (System ARchitects Apprentice): Modeling, analysis, and simulation support for design of concurrent systems , 1986, IEEE Transactions on Software Engineering.

[120]  Brian J. N. Wylie,et al.  Annai: An Integrated Parallel Programming Environment for Multicomputers , 1996 .

[121]  Mukesh Singhal,et al.  Performance Analysis of the Communication Architecture of the Connection Machine , 1992, IEEE Trans. Parallel Distributed Syst..

[122]  Frederic A. Van-Catledge.  Toward a General Model for Evaluating the Relative Performance of Computer Systems , 1989, Int. J. High Perform. Comput. Appl..

[123]  Michael D. Rice,et al.  Modeling the Serial and Parallel Fractions of a Parallel Algorithm , 1991, J. Parallel Distributed Comput..

[124]  Alex Kapelnikov,et al.  A Modeling Methodology for the Analysis of Concurrent Systems and Computations , 1989, J. Parallel Distributed Comput..

[125]  Ko-Yang Wang.  Precise compile-time performance prediction for superscalar-based computers , 1994, PLDI '94.

[126]  Ilya Gertner,et al.  Non-intrusive and interactive profiling in parasight , 1988, PPEALS '88.

[127]  Rafael Hector Saavedra-Barrera,et al.  CPU performance evaluation and execution time prediction using narrow spectrum benchmarking , 1992 .

[128]  Francine Berman,et al.  Predicting the performance of large programs on scalable multicomputers , 1992, Proceedings Scalable High Performance Computing Conference SHPCC-92..

[129]  V. Sarkar,et al.  Automatic partitioning of a program dependence graph into parallel tasks , 1991, IBM J. Res. Dev..

[130]  Manish Gupta,et al.  Compile-time estimation of communication costs on multicomputers , 1992, Proceedings Sixth International Parallel Processing Symposium.

[131]  Randolph D. Nelson,et al.  A performance evaluation of a general parallel processing model , 1990, SIGMETRICS '90.

[132]  Stephen D. Roberts.  Modeling and simulation with INSIGHT , 1986, WSC '86.

[133]  Reda A. Ammar,et al.  Micro Time Cost Analysis of Parallel Computations , 1991, IEEE Trans. Computers.

[134]  Günter Haring,et al.  Computer Performance Evaluation, Modeling Techniques and Tools, 7th International Conference, Vienna, Austria, May 3-6, 1994, Proceedings , 1994, Computer Performance Evaluation.

[135]  Thomas Fahringer,et al.  The Weight Finder - An Advanced Profiler for Fortran Programs , 1994, Automatic Parallelization.

[136]  Connie U. Smith,et al.  Software Performance Engineering: A Case Study Including Performance Comparison with Design Alternatives , 1993, IEEE Trans. Software Eng..

[137]  K. Baclawski,et al.  A network emulation tool , 1987, Simulation of Computer Networks.

[138]  Dennis Gannon,et al.  Building analytical models into an interactive performance prediction tool , 1989, Proceedings of the 1989 ACM/IEEE Conference on Supercomputing (Supercomputing '89).

[139]  David Ben-Arieh,et al.  Towards a knowledge-based network simulation environment , 1985, WSC '85.

[140]  P. Krystosek,et al.  REN: a reconfigurable experimental network , 1987 .

[141]  Thomas Fahringer,et al.  Automatic performance prediction to support parallelization of Fortran programs for massively parallel systems , 1992, ICS '92.

[142]  Mark J. Clement,et al.  Analytical performance prediction on multicomputers , 1993, Supercomputing '93. Proceedings.

[143]  Cherri M. Pancake,et al.  Software Support for Parallel Computing: Where Are We headed? , 1991 .

[144]  Wolfgang E. Nagel,et al.  Optimization of Output Bandwidth from an Intel Paragon , 1994 .

[145]  B.J.N. Wylie,et al.  Tool-supported parallel application development , 1996, Conference Proceedings of the 1996 IEEE Fifteenth Annual International Phoenix Conference on Computers and Communications.

[146]  Domenico Ferrari,et al.  A general-purpose software measurement tool , 1974, SIGMETRICS '74.

[147]  Vivek Sarkar,et al.  Partitioning and scheduling parallel programs for execution on multiprocessors , 1987 .

[148]  Vivek Sarkar,et al.  Determining average program execution times and their variance , 1989, PLDI '89.

[149]  Marina C. Chen,et al.  Compiling Communication-Efficient Programs for Massively Parallel Machines , 1991, IEEE Trans. Parallel Distributed Syst..

[150]  Krishna R. Pattipati,et al.  CAPRES: a software tool for modeling and analysis of fault-tolerant computer architectures , 1989, Conference Proceedings., IEEE International Conference on Systems, Man and Cybernetics.

[151]  Jacques Cohen,et al.  Computer-assisted microanalysis of programs , 1982, CACM.

[152]  C. Murray Woodside,et al.  A Three-View Model for Performance Engineering of Concurrent Software , 1995, IEEE Trans. Software Eng..

[153]  Josep Torrellas,et al.  Analysis of Critical Architectural and Program Parameters in a Hierarchical Shared Memory Multiprocessor , 1990, SIGMETRICS.

[154]  J.-P. Prost,et al.  A methodology and a tool for parallel systems performance analysis , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.