Data-Parallel Load Balancing Strategies

Abstract Programming irregular and dynamic data-parallel algorithms must consider the effect of data distribution. The implementation of a load balancing algorithm is quite a difficult task for the programmer. However, a load balancing strategy may be developed independently of the application. The integration of such a strategy into the data-parallel algorithm may be relevant to a library or a data-parallel compiler run-time. We propose load distribution data-parallel algorithms for a class of irregular data-parallel algorithms called stack algorithms. Our algorithms allow the use of regular and/or irregular communication patterns to exchange the works between processors. The results of theoretical analysis of these algorithms are presented. They allow different load balancing algorithms to be compared and the identification of criteria to choose between them.

[1]  Cyril Fonlupt,et al.  Analysis of Synchronous Dynamic Load Balancing Algorithms , 1995, PARCO.

[2]  W. Daniel Hillis,et al.  The connection machine , 1985 .

[3]  Reinhard Lüling,et al.  Load balancing in large networks: a comparative study , 1991, Proceedings of the Third IEEE Symposium on Parallel and Distributed Processing.

[4]  Rice UniversityCORPORATE,et al.  High performance Fortran language specification , 1993 .

[5]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[6]  George Cybenko,et al.  Dynamic Load Balancing for Distributed Memory Multiprocessors , 1989, J. Parallel Distributed Comput..

[8]  V. A. Saletore.  A Distributed and Adaptive Dynamic Load Balancing Scheme for Parallel Processing of Medium-Grain Tasks , 1990, Proceedings of the Fifth Distributed Memory Computing Conference, 1990..

[9]  Isaac D. Scherson,et al.  An analysis of diffusive load-balancing , 1994, SPAA '94.

[10]  David M. Nicol,et al.  Rectilinear Partitioning of Irregular Data Parallel Computations , 1994, J. Parallel Distributed Comput..

[11]  Francis C. M. Lau,et al.  The Generalized Dimension Exchange Method for Load Balancing in k-ary n Cubes and Variants , 1995, J. Parallel Distributed Comput..

[12]  Reuven Y. Rubinstein,et al.  Simulation and the Monte Carlo method , 1981, Wiley series in probability and mathematical statistics.

[13]  Wentong Cai,et al.  How network topology affects dynamic loading balancing , 1996, IEEE Parallel Distributed Technol. Syst. Appl..

[14]  Bruce M. McMillin,et al.  Experimental Comparison Of Bidding And Drafting Load Sharing Protocols , 1990, Proceedings of the Fifth Distributed Memory Computing Conference, 1990..

[15]  Julia M. Goodfellow,et al.  Parallelization strategies for molecular simulation using the Monte Carlo algorithm , 1993, J. Comput. Chem..

[16]  Cyril Fonlupt.  Distribution dynamique de données sur machine SIMD , 1994 .

[17]  Richard E. Korf,et al.  Parallel tree search on a SIMD machine , 1991, Proceedings of the Third IEEE Symposium on Parallel and Distributed Processing.

[18]  David M. Nicol,et al.  Improved Algorithms for Mapping Pipelined and Parallel Computations , 1991, IEEE Trans. Computers.

[19]  Cyril Fonlupt,et al.  A Data-Parallel View of the Load Balancing - Experimental Results on MasPar MP-1 , 1994, HPCN.

[20]  Vipin Kumar,et al.  Unstructured tree search on SIMD parallel computers: a summary of results , 1992, Proceedings Supercomputing '92.

[21]  Richard E. Korf,et al.  Depth-First Heuristic Search on a SIMD Machine , 1993, Artif. Intell..

[22]  W. Daniel Hillis,et al.  Data parallel algorithms , 1986, CACM.

[23]  Jean-Luc Dekeyser,et al.  Supporting Irregular and Dynamic Computations in Data Parallel Languages , 1996, The Data Parallel Programming Model.

[24]  Z. C.,et al.  Analysis of The Generalized Dimension Exchange Method forDynamic Load Balancing , 1992 .

[25]  Michael Metcalf,et al.  Fortran 90 Explained , 1990 .

[26]  Luc Bougé.  The Data Parallel Programming Model: A Semantic Perspective , 1996, The Data Parallel Programming Model.

[27]  Walter S. Brainerd,et al.  Programmer's guide to Fortran 90 , 1990 .

[28]  Anthony P. Reeves,et al.  Strategies for Dynamic Load Balancing on Highly Parallel Computers , 1993, IEEE Trans. Parallel Distributed Syst..

[29]  Yves Robert,et al.  Elastic Load-Balancing for Image Processing Algorithms , 1991, ACPC.