A Communications-Oriented Methodology for Load Balancing in Parallel Relational Query Processing

The increasing amount of data manipulated in very large databases 1 and the complexity of the queries to be processed makes parallelism appear as one of the most promising research axes for future databases applications. Load balancing for relational operators has been widely studied and join algorithms including dynamic load balancing capabilities have been proposed 1]. However, for whole relational queries this problem still remains largely open 2]. We propose an original dynamic load balancing technique based on an elastic approach as solution. In our approach, an input query is rst transformed into a plan to be executed on the parallel machine. Then, the number of processors to execute every operator is determined, and a mapping algorithm allocates the physical processors. For complex queries this might not be suucient, as some queries do not allow a correct prediction of the evaluation of relations cardinality. Furthermore data can be non-uniformly distributed over the processors (data-skew). So a control mechanism has to trace the query processing. A special hierarchy of supervisors is introduced to interfere in case of load imbalance or estimation errors. Control points in the query triggers, if necessary, dynamic load balancing and rescheduling of the remaining query. Dynamic load balancing is achieved with the help of elastic redistribution algorithm. Data are then redistributed in a order to minimize the data movements and to keep the locality of the relations. Tests have been performed on the iPSC860 of Intel, using a real database of french medicaments with about 10000 entries. The performance evaluation focused on the measurement of achieved speed-up and load balance in query processing. Thus for 8 processors we had a speed-up of nearly 7 for all the tested queries. Load balance was evaluated by comparing the diierent execution times of processors working on the same operator. The variance of the diierent times was for the worst cases under 2% of the whole execution time. The performance results connrmed so the robustness of our theoretical approach, specially good load balancing was achieved with the proposed methods. 1 Terabyte databases containing billions of records are no more exotic.