Architecture and Early Performance of the New IBM HPS Fabric and Adapter

In this paper we describe the architecture, design, and performance of the new cluster switch fabric and adapter called HPS (High Performance Switch). HPS delivers very low latency and very high bandwidth. We demonstrate latency of less than 4.3us MPI library; 1.8GB/s of delivered unidirectional bandwidth and 2.9GB/s of bidirectional bandwidth between 2 MPI tasks running on 1.9GHz Power 4+ IH based nodes. HPS also supports RDMA (remote direct memory access capability). A unique capability of RDMA over HPS is that reliable RDMA is supported over an underlying unreliable transport (unlike Infiniband and other RDMA transport protocols which depend on the underlying transport being reliable). We profile the performance of RDMA and its impact on striping for systems in which multiple network adapters are available to tasks of parallel jobs.