Parallel Communication Mechanisms for Sparse, Irregular Applications

Parallel systems are becoming a significant computing technology, not only for high performance computing, but also for commodity servers. The goal of this research is to identify core communication mechanisms which both exploit architectural trends and support real applications. We demonstrate that cache-coherent shared memory hardware is such a core mechanism, even for applications with little data re-use and data-driven synchronization. This thesis makes three major contributions. First, we perform an in-depth study of the interaction between communication mechanisms and sparse, irregular applications. Second, we present the Remote Queues (RQ) communication model, an abstraction which synthesizes more efficient synchronization for hardware-supported shared memory and other complex systems. Third, we characterize the relative performance of all of our mechanisms as processor speed and machine size scale. On the MIT Alewife Multiprocessor, we find that shared memory provides high performance with lower code complexity than message passing on our irregular problems. This is primarily due to four reasons. First, a 5-to-1 ratio between global and local cache misses makes memory copies in bulk communication expensive relative to communication via shared memory. Second, although message passing has synchronization semantics superior to shared memory for data-driven computation, efficient shared memory can overcome this handicap by using global read-modify-writes to change from the traditional owner-computes model to a producer-computes model. Third, the Remote Queues communication model generalizes such a change, providing the semantics and performance of polling active messages on a wide variety of systems. Fourth, bulk transfers can result in high processor idle times in irregular applications. Finally, we characterize multiprocessor design points where message passing and bulk transfer can perform better than shared memory. In particular, we find that shared memory uses more than four times the network bandwidth as message passing. Unless an application’s performance is already limited by local memory speeds, network bandwidth and latency threaten to become a serious problem. Our study indicates that machines based on modern microprocessors, such as the Cray T3E, must resort to expensive, high-dimensional networks to support shared-memory traffic. Furthermore, the round-trip nature of shared memory may not be able to tolerate the latencies of future networks.