Assessing Single-Message and Multi-Node Communication Performance of InfiniBand

We present a micro benchmark suite to evaluate InfiniBandtrade implementations with regards to single message performance and the addressing of many hosts. We use a 1:n communication pattern to assess the latency and bandwidth for all different combinations of InfiniBandstrade transport services and functions. The results gathered in this study are used to optimize MPI collective communication operations where 1:n communication schemes are not used widely today. We show that applications as well as collective algorithms can benefit from sending multiple messages in a single round. Moreover, the results are used to choose the transport service and function to develop InfiniBandtrade optimized collective communication functions. Our study compares all available transport options and shows that single packet sends can be very expensive compared to multi packet sends