Impact of InfiniBand DC Transport Protocol on Energy Consumption of All-to-All Collective Algorithms

Data intensive collective operations have a notable impact on the execution time and consequently the energy consumption of HPC applications owing to the amount of memory/processor/network resources involved in the data movement. However, mechanisms such as offload and one-sided transfers that are backed by RDMA-enabled interconnects like InfiniBand along with modern transport protocols like Dynamic Connected (DC) provide new ways to express collective communication. Despite this, researchers have not considered the use of an RDMA-based algorithm backed by the appropriate transport protocol for reducing energy consumption of collective operations. In this paper, we take up this challenge and study the impact that RDMA and transport protocol aware designs can have on the processor/memory energy and performance of dense collective operations like All-to-all. Through evaluation, we also identify that while a single transport protocol may bring both performance and energy benefits for one application it may not do so consistently for all applications. Motivated by this, we propose designs that yield both benefits for all evaluated applications. Experimental evaluation shows that our proposed designs are able to deliver up to 1.7X savings in energy with little or no degradation in the communication performance for All-to-all collective operations on modern HPC systems.