A new design of RDMA-based small message channels for InfiniBand clusters

We propose a novel design for RDMA-based small message channels that significantly improves the MVAPICH design. First, we develop a technique that eliminates persistent buffer association, a scheme used in MVAPICH that not only results in significant memory requirement, but also imposes restrictions in memory management. Building upon this technique, we propose a novel shared RDMA-based small message channel design that allows MPI processes on the same SMP node to share small message channels, which greatly reduces the number of small message channels needed for an MPI program on clusters with SMP nodes. Our techniques considerably improve the scalability and reduce memory requirement in comparison to MVAPICH, allowing RDMA-based small message channels to be used by a much larger number of MPI processes. The experimental results demonstrate that our techniques achieve the improvements without adding noticeable overheads or sacrificing the performance benefits of RDMA in practice.