Data Distribution , Migration and Replication on a cc-NUMA Architecture

It is well known that, although cc-NUMA architectures allow construction of large scale shared memory systems, they are more difficult to program effectively because data locality is an important consideration. Support for specifying data distribution in OpenMP has been the subject of much debate [1], [4], and several proposed implementations. These take the form of data distribution directives, giving the programmer control of where data is placed in the memory system. In the absence of additional directives, data distribution can be controlled by exploiting the system’s allocation policy. In most cc-NUMA systems, data is placed on the node which first accesses it: the so-called first touchpolicy. An alternative strategy is to give this control not to the programmer but to the operating system, by allowing the location of data in memory to change as a program executes. This can be done either by data migration, where pages can move between nodes, but there is only ever one copy, or by replication, where multiple copies of pages can exist. In this study, we examine the interactions between data distribution (implemented via the first touch policy), migration, and replication on a prototype cc-NUMA architecture. On this system it appears that replication is almost always more effective that migration, despite the additional cost in memory usage. Data distribution can be effective in applications where there is obviously a “correct” distribution. In many applications the “correct” distribution is not so obvious, and although it pays to distribute data in the absence of migration and replication, some combination of replication and migration can often achieve comparable performance.