Performance modeling and analysis of correlated parallel computations

A performance analysis methodology for correlated parallel computations based on statistical theory is proposed. Divide-and-conquer strategy is widely used in solving problems in parallel by partitioning and allocating a number of given tasks to available computing resources. When the tasks exhibit run-time-dependent behaviors during execution and share a universal distribution function in their execution times, analysis of parallel execution time can be performed with the assistance of probabilistic and statistical models. Correlation (dependence) in execution times among tasks has posed a significant factor in influencing the analysis accuracy which is unmanageable by any known analysis methodologies. We establish a relation between a task's or a processor's execution time and the parallel execution time, in terms of expected value as well as variance when each task's execution time can be closely modeled by a normal distribution, for either uncorrelated or correlated tasks. This relation is then applied to the modeling and analysis of various parallel computation paradigms in which different communication and synchronization patterns along the processing are present. The method proposed has a wider application scope and gives more accurate prediction results than previously known approaches. We also show that, as an extended application of the analysis method to a large scope of problems, load balance among processors can be vastly improved with some novel static task allocation technique in manipulating the correlation among tasks. Experimental results in analyzing a parallel tree search algorithm and two parallel sorting algorithms show very accurate analysis and prediction with the proposed method.