Runtime Optimization of Broadcast Communications Using Dynamic Network Topology Information from MPI

Modern commodity compute clusters are often composed of many multi-core nodes, that are connected via a network to each other. On multi-core clusters, inter-node network communications are typically an order of magnitude slower than those between processes on the same node, which effectively creates a heterogeneous, tiered network topology. Presently, most MPI implementations assume a homogeneous network composition, which causes them to have less than optimal performance on multi-core clusters. In this paper, we treat a multi-core cluster as a heterogeneous cluster and optimize the performance of MPI broadcast communications by scheduling messages according to topology information. We experimentally demonstrate that previous heuristics for heterogeneous clusters such as Fastest Edge First (FEF) do not produce optimal results on multi-core clusters for broadcast communications. Our solution is to modify the Fastest Edge First heuristic by imposing an additional constraint, that permits only one core per node to participate in inter-node communications, creating a nested binomial tree structure. Using this constraint we are able to achieve performance gains of 20%-60% over the MPI broadcast implementation on homogeneous, multi-core clusters.