The influence of random delays on parallel execution times

Stochastic models are widely used for the performance evaluation of parallel programs and systems. The stochastic assumptions in such models exe intended to represent non-deterministic processing requirements as well as random delays due to inter-process communication end resource contention. In this paper, we provide compelling analytical and experimental evidence that in current and foreseeable shared-memory programs, communication delays introduce negligible variance into the execution time between synchronization points. Furthermore, we show using direct measurements of variance that other sources of randomness, particularly non-deterministic computational requirements, also do not introduce significant variance in many programs. We then use two examples to demonstrate the implications of these results for parallel program performance prediction models, as well as for general stochastic models of parallel systems.

[1]  Satish K. Tripathi,et al.  An analysis of several processor partitioning policies for parallel computers , 1991 .

[2]  Alex Kapelnikov,et al.  A Modeling Methodology for the Analysis of Concurrent Systems and Computations , 1989, J. Parallel Distributed Comput..

[3]  Anne Greenbaum.  Synchronization costs on multiprocessors , 1989, Parallel Comput..

[4]  Vivek Sarkar,et al.  Determining average program execution times and their variance , 1989, PLDI '89.

[5]  Robert E. Tarjan,et al.  An Efficient Parallel Biconnectivity Algorithm , 2011, SIAM J. Comput..

[6]  Kishor S. Trivedi.  Probability and Statistics with Reliability, Queuing, and Computer Science Applications , 1984 .

[7]  Asser N. Tantawi,et al.  The Order Statistics of the Sojourn Times of Customers that Form a Single Batch in the Mx/M/c Queue , 1989 .

[8]  AlgorithmsGary Lewandowski,et al.  Realistic Analysis of Parallel Dynamic Programming , 1992 .

[9]  Anoop Gupta,et al.  SPLASH: Stanford parallel applications for shared-memory , 1992, CARN.

[10]  Alexander Thomasian,et al.  Analytic Queueing Network Models for Parallel Processing of Task Systems , 1986, IEEE Transactions on Computers.

[11]  François Baccelli,et al.  On the execution of parallel programs on multiprocessor systems—a queuing theory approach , 1990, JACM.

[12]  Yuris O. Fuentes,et al.  Parallel computational microhydrodynamics : communication scheduling strategies , 1992 .

[13]  Ronald W. Wolff,et al.  Stochastic Modeling and the Theory of Queues , 1989 .

[14]  Alan Weiss,et al.  Allocating Independent Subtasks on Parallel Processors , 1985, IEEE Transactions on Software Engineering.

[15]  Asser N. Tantawi,et al.  Performance Analysis of Parallel Processing Systems , 1988, IEEE Trans. Software Eng..

[16]  Terry Williams,et al.  Probability and Statistics with Reliability, Queueing and Computer Science Applications , 1983 .

[17]  Mary K. Vernon,et al.  The performance of multiprogrammed multiprocessor scheduling algorithms , 1990, SIGMETRICS '90.

[18]  C. S. Chang,et al.  Bounds on the speedup and efficiency of partial synchronization in parallel processing systems , 1995, JACM.

[19]  James R. Larus,et al.  The Wisconsin Wind Tunnel: virtual prototyping of parallel computers , 1993, SIGMETRICS '93.

[20]  Randolph D. Nelson,et al.  A performance evaluation of a general parallel processing model , 1990, SIGMETRICS '90.

[21]  John Zahorjan,et al.  Processor scheduling in shared memory multiprocessors , 1990, SIGMETRICS '90.

[22]  V. Joseph Subash Mohan,et al.  Performance of parallel programs: model and analyses , 1984 .

[23]  S. Madala,et al.  Performance of Synchronous Parallel Algorithms with Regular Structures , 1991, IEEE Trans. Parallel Distributed Syst..

[24]  Michel Dubois,et al.  Performance of Synchronized Iterative Processes in Multiprocessor Systems , 1982, IEEE Transactions on Software Engineering.

[25]  Eugene D. Brooks.  The indirect k-ary n-cube for a vector processing environment , 1988, Parallel Comput..

[26]  Stephen F. Lundstrom,et al.  Predicting Performance of Parallel Computations , 1990, IEEE Trans. Parallel Distributed Syst..