Implementing TreadMarks over GM on Myrinet: challenges, design experience, and performance evaluation

Software based DSM systems like TreadMarks have traditionally not performed well compared to message passing applications because of the high overhead of communication associated with traditional stack based protocols like UDP. Modern interconnects like Myrinet offer reliable message delivery with very low communication overhead through user level protocols. This paper examines the viability of implementing a thin communication substrate between TreadMarks and Myrinet GM, the rationale being that a layer tuned to the needs of the application would offer better performance and scalability as opposed to a generic UDP layer. Trade-offs for various design alternatives for buffer management, connection setup, advance posting of descriptors and asynchronous messages are discussed. We have implemented the best of these strategies in a layer that is bound to TreadMarks at compile time. Results from micro-benchmarks and applications show that not only does the specialized implementation perform better, it also exhibits better parallel speedup and scalability. A reduction in total application execution time of up to a factor of 6.3 for a 16 node system is demonstrated in comparison with the original implementation. The implementation also exhibits superior scaling properties as the application size is increased.