Acceleration for MPI derived datatypes using an enhancer of memory and network

This paper presents a support function for MPI derived datatypes on an enhancer of memory and network named DIMMnet-3. It is a network interface with vector access functions and multi-banked extended memory, which is under development. Semi-hardwired derived datatype communication based on RDMA with hardwired scatter and gather is proposed. This mechanism and MPI using it are implemented and validated on DIMMnet-2 which is a former prototype operating on DDR DIMM slot. The performance of scatter and gather transfer of 8byte elements with large interval by using vector commands of DIMMnet-2 is 6.8 compared with software on a host. Proprietary benchmark of MPI derived datatype communication for transferring a submatrix corresponding to a narrow HALO area is executed. Observed bandwidth on DIMMnet-2 is far higher than that for similar condition with VAPI based MPI implementation on InfniBand, even though very old generation FPGA, poorer CPU and motherboard are used. This function will avoid cache pollution and save CPU time for processing with local data which can be overlapped with communication. A new commercial machine with vector scatter/gather functions in NIC named SGI Altix UV is launched recently. It may be able to adopt our proposed concept partially, even though the capacity and fine grain access throughput of main memory attached with CPU are not enhanced on it.