Fine tuning network MTU for HPC cluster: Correlating MTU with PCIe parameters and HCA architecture

An Efficient Network is an important component of a cluster based parallel computing. PCI Express based Host Channel Adapter (or Network Interconnect) is the most commonly used network interface in the clusters and supercomputers. The system memory reads and writes that HCA performs are highly constrained by PCI Express protocol. Similarly, the HCA architecture plays an important role in deciding network performance, as it has its own scheduling or queuing mechanism. Therefore, it would be beneficial to take into account the Host communication protocol and the Network architecture while tuning the Network parameters. In this paper, we varied one of the Network parameters: the network transfer size i.e. Maximum Transfer Unit (MTU), and observed its effect on the Network performance. We also discuss the PCI Express´ parameters as well as the HCA architecture that can be considered for deciding the optimal value of MTU. Based on the experimental observations and analysis, we conclude that the most suitable value of MTU is 1024 Bytes with the HCA architecture discussed in the paper.