Developing an OpenSHMEM Model Over a Switchless PCIe Non-Transparent Bridge Interface

OpenSHMEM is an emerging parallel programming model in distributed systems that supports partitioned global address space (PGAS). It was originally developed for Cray systems, and it was subsequently adopted by numerous vendors of the InfiniBand and Ethernet network interfaces, since most high-performance computing (HPC) systems deploy switch-based interconnect networks using InfiniBand and Ethernet technologies. Recently, Peripheral Component Interconnect Express (PCIe) has become one of the most promising candidates for deploying cost-effective popular HPC systems because of its low cost and powerful features, as well as the Non-Transparent Bridge(NTB) technology (interconnect interface for PCIe). However, there is little work done on implementing the OpenSHMEM library for PCIe systems. Herein, we introduce a prototype of a switchless interconnect network with PCIe NTB. In our switchless interconnect network system, the computing nodes are interconnected via the PCIe NTB interface. Based on the PCIe NTB interconnect network, we implemented the OpenSHMEM programming interface to support the PGAS mechanism for PCIe NTB interconnect networks. Our design and implementation of an OpenSHMEM programming model via PCIe NTB shows that is feasible and possible to have a PCIe-based, cost-effective, high performance interconnect network in a high-performance programming model.