Dynamic Load Balancing of Parallel Computational Iterative Routines on Highly Heterogeneous HPC Platforms

Traditional load balancing algorithms for data-intensive iterative routines can successfully load balance relatively small problems. We demonstrate that they may fail on highly heterogeneous HPC platforms. Traditional algorithms use models of processors' performance which are too simplistic to reflect the many aspects of heterogeneity. This paper presents a new class of dynamic load balancing algorithms based on the advanced functional performance models. The models are functions of problem size and are built adaptively by measuring the execution time of each iteration. Two particular load balancing algorithms of this class are presented in the paper. The low execution cost of distribution of computations between heterogeneous processors in these algorithms make them suitable for employment in self-adaptable applications. Experimental results demonstrate that our algorithms can successfully balance data-intensive iterative routines on parallel platforms with high heterogeneity for the whole range of problem sizes.

[1]  Debasish Ghose,et al.  Divisible Load Theory: A New Paradigm for Load Scheduling in Distributed Systems , 2004, Cluster Computing.

[2]  Robert Higgins,et al.  Modelling the Performance of Processors in Heterogeneous Computing Environments , 2011 .

[3]  Philip Rabinowitz,et al.  Numerical methods for nonlinear algebraic equations , 1970 .

[4]  Alexey L. Lastovetsky,et al.  Distributed Data Partitioning for Heterogeneous Processors Based on Partial Estimation of Their Functional Performance Models , 2009, Euro-Par Workshops.

[5]  Jeanette P. Schmidt,et al.  Load-sharing in heterogeneous systems via weighted factoring , 1996, SPAA '96.

[6]  Antonio J. Plaza,et al.  Automatic tuning of iterative computation on heterogeneous multiprocessors with ADITHE , 2011, The Journal of Supercomputing.

[7]  Francisco Almeida,et al.  Dynamic Load Balancing on Dedicated Heterogeneous Systems , 2008, PVM/MPI.

[8]  Mohammed J. Zaki,et al.  Compile-Time Scheduling Algorithms for a Heterogeneous Network of Workstations , 1997, Comput. J..

[9]  Ioana Banicescu,et al.  Dynamic load balancing with adaptive factoring methods in scientific applications , 2007, The Journal of Supercomputing.

[10]  Yves Robert,et al.  Mapping and load-balancing iterative computations , 2004, IEEE Transactions on Parallel and Distributed Systems.

[11]  George Cybenko,et al.  Dynamic Load Balancing for Distributed Memory Multiprocessors , 1989, J. Parallel Distributed Comput..

[12]  Hiroshi Akima,et al.  A New Method of Interpolation and Smooth Curve Fitting Based on Local Procedures , 1970, JACM.

[13]  Shinji Yamashita,et al.  Static Load Balancing of Parallel PDE Solver for Distributed Computing Environment , 2000 .

[14]  Jacques M. Bahi,et al.  Dynamic load balancing and efficient load estimators for asynchronous iterative algorithms , 2005, IEEE Transactions on Parallel and Distributed Systems.

[15]  Alexey L. Lastovetsky,et al.  Data Partitioning with a Functional Performance Model of Heterogeneous Processors , 2007, Int. J. High Perform. Comput. Appl..

[16]  Mjd Powell,et al.  A Fortran subroutine for solving systems of non-linear algebraic equations , 1968 .

[17]  Xiang-Yang Li,et al.  Dynamic Load Balancing for Parallel Adaptive Mesh Refinement , 1998, IRREGULAR.

[18]  Alexey L. Lastovetsky,et al.  Building the functional performance model of a processor , 2006, SAC.