Efficient load balancing and data remapping for adaptive grid calculations

ABSTRACTMesh adaption is a powerful tool for efficient unstructured-grid computations but causes load imbalance among proces-sors on a parallel machine. We present a novel method todynamically balance the processor workloads with a globalview. This paper presents, for the first time, the imple-mentation and integration of all major components withinour dynamic load balancing strategy for adaptive grid cal-culations. Mesh adaption, repartitioning, processor assign-ment, and remapping are critical components of the frame-work that must be accomplished rapidly and efficiently soas not to cause a significant overhead to the numerical sim-ulation. Previous results indicated that mesh repartitioningand data remapping are potential bottlenecks for perform-ing large-scale scientific calculations. We resolve these issuesand demonstrate that our framework remains viable on alarge number of processors.1 INTRODUCTIONDynamic mesh adaption on unstructured grids is a power-ful tool for computing unsteady three-dimensional problemsthat require grid modifications to efficiently resolve solu-tion features. By locally refining and coarsening the meshto capture flowfield phenomena of interest, such proceduresmake standard computational methods more cost effective.Highly refined meshes are required to accurately captureshock waves, contact discontinuities, vortices, and shear lay-ers. Local mesh adaption provides the opportunity to obtainsolutions that are comparable to those obtained on globally-refined grids but at a much lower cost.Unfortunately, the adaptive solution of unsteady prob-lems causes load imbalance among processors on a paral-lel machine. This is because the computational intensity isboth space and time dependent. An efficient parallel imple-mentation of such methods is extremely difficult to achieve,primarily because of the dynamically-changing nonuniformgrid. Various methods on dynamic load balancing have beenreported to date [5-9,11-14,16-18,24-26]; however, most ofthem either lack a global view of loads across processors orProceedings of the 9th ACM Symposium on Parallel Al-gorithms and Architectures, Newport, Rhode Island, June22-25, 1997.do not apply their techniques to realistic large-scale appli-cations.Figure 1 depicts our framework for parallel adaptive flowcomputation. It consists of a flow solver and a mesh adap-tor, with a partitioner and a remapper that load balancesand redistributes the computational mesh when necessary.Our goal is to build a portable system for efficiently per-forming adaptive large-scale flow calculations in a parallelmessage-passing environment. The mesh is first partitionedand mapped among the available processors. The flow solverthen runs for several iterations, updating solution variables.Once an acceptable solution is obtained, the mesh adaptionprocedure is invoked. It first targets edges for coarseningand refinement based on an error indicator computed fromthe flow solution. The old mesh is then coarsened, resultingin a smaller grid. Since edges have already been marked forrefinement, it is possible to exactly predict the new meshbefore actually performing the refinement step. Programcontrol is thus passed to the load balancer at this time. Aquick evaluation step determines if the new mesh will be sounbalanced as to warrant a repartitioning. If the currentpartitions will remain adequately load balanced, control mpassed back to the subdivision phase of the mesh adaptor.Otherwise, a repartitioning procedure is used to divide thenew mesh into subgrids. The new partitions are then reas-signed to the processors in a way that minimizes the costof data movement. If the remapping cost is less than thecomputational gain that would be achieved with balancedpartitions, all necessary data is appropriately redistributed.Otherwise, the new partitioning is discarded. The computa-tional mesh is then actually refined and the flow calculationis restarted.Notice from the framework in Fig. 1 that splitting themesh refinement step into two distinct phases of edge mark-ing and mesh subdivision allows the subdivision phase tooperate in a more load balanced fashion. In addition, sincedata remapping is performed before the mesh grows in sizedue to refinement, a smaller volume of data is moved. Thiscan lead to a potentially significant savings in the redistribu-tion cost. The load balancer also balances the computationalload for the flow solver while reducing the runtime commu-nication. This is important because flow solvers are usuallyseveral times more expensive than mesh adaptors. In anycase, it is obvious that mesh adaption, repartitioning, pro-cessor assignment, and remapping are critical componentsof the framework and must be accomplished rapidly and ef-ficiently so as not to cause a significant overhead to the flowcomputation.