Dynamic Setup for Clusters with Multi-master Architecture

The paper deals with the problem of multi-master setup for clusters running PROOF, which is a master-worker based framework used at CERN, preferably for analysis of high energy physics data. The goal is to determine an optimal number of masters for the given task in order to make the task execution time as short as possible. Based on the analysis of PROOF processing work-flow, the merger-based algorithm is designed and implemented. It introduces a concept of the merger, which is a node acting as a worker during the computation phase and as a master during the final phase of sub-results merging. The number and selection of merger nodes is performed dynamically, and depends both on the cluster size and its recent performance. The performance of the merger-based algorithm is compared to the standard merging approach.