RDMA-based and SMP-aware Multi-port All-Gather on Multi-rail QsNet^II SMP Clusters

Clusters of symmetric multiprocessors (SMP) are more commonplace than ever in achieving high- performance. Scientific applications running on clusters employ collective communications extensively. Using shared memory communication among co- located processes on SMP nodes as well as remote direct memory access (RDMA) operations for inter- node communication and trying to overlap them is a proven technique in boosting the performance of collective operations. The effect is much more pronounced when efficient multi-port collectives on multi-rail networks are devised and implemented. In this work, we design and implement multi-port RDMA-based and SMP-aware all-gather algorithms with message striping over multi-rail QsNe^II directly at the Elan level. We compare our algorithms against RDMA-only traditional algorithms and the native elan_gather(). Our performance results indicate that the proposed SMP-aware Brack all-gather gains an improvement of up to 1.96 for 4KB messages over the native elanjgather(). Meanwhile, the direct algorithm achieves up to 1.49 improvement for 32 KB messages.