OpenMP vs MPI for Ocean Models

Two ocean models, NLOM and NCOM (NRL Layered and Coastal Ocean Models), use domain decomposition to obtain scalability that has been demonstrated on up to 1,152 nodes (Wallcraft and Moore, 1997; Parallel Comp.). In both models, communication between nodes is for the most part performed by calling an application specific programming interface (ASPI), i.e. a set of Fortran subroutines that perform pre-defined high-level communication operations. This allows different low-level “message passing” APIs to be added with relatively little effort. The first two supported APIs were MPI and the SHMEM put/get library, with selection between them being made at compile time via cpp macros. Domain decomposition is a special case of Single Program Multiple Data (SPMD) parallelization where each node “owns” a distinct sub-domain. When implemented using message passing, each node only holds in memory its own sub-domain plus a large enough “halo” (or set of “ghost cells”) containing copies of values from nearby nodes to allow almost all computation to proceed locally. This splits the problem into a communication phase (updating the halo) followed by a computation phase (with no communication involved). Real applications must handle the additional complexity of I/O and global operations caused by distributing the region across the nodes, but halo exchange is the heart of domain decomposition.