High Performance RDMA-based Multi-port All-gather on Multi-rail QsNet II

Scientific applications written in MPI use collective communications intensively. Efficient and scalable implementation of such collective operations is therefore crucial to the performance of MPI applications running on clusters. Quadrics QsNet^II is a high-performance network that implements some collectives at its Elan user-level library. Its MPI implementation uses such primitives directly. Quadrics communication software supports point-to-point message striping over multi-rail QsNet^II networks. However, multi-rail collectives, other than broadcast, are not supported. In this work, we propose, design and implement a number of RDMA- based multi-port algorithms for the all-gather operation over multi-rail QsNet^II clusters directly at the Elan level. Our performance results indicate that the proposed multi-port all-gather Direct algorithm gains an improvement of up to 1.49 for 32 KB messages over the native Elan_gather() collective.