Modeling Many-Task Computing Workloads on a Petaflop IBM Blue Gene/P Supercomputer

Understanding the behavior of Bag-of-Tasks (BOT) is crucial for analyzing workflow-generated Many-Task Computing (MTC) workloads to aid in designing optimized job scheduling systems. Future job scheduling systems will need to be able to schedule large bags of tasks onto large-scale supercomputers and adaptive clouds with heterogeneous processors, I/O performance, and cost, all while minimizing job turn-around time and respecting the upper bound for the user-defined budget. Due to the strong periodicity and self-similarity during long time periods, BOTs have been shown to be an efficient approach for modeling High-Throughput Computing (HTC) workloads. However, applying the same analysis to MTC workloads poses significant challenges due to the significantly larger scale in terms of number of tasks, resource usage, and work granularity. In this paper, we extract two workloads from traces obtained from running MTC applications on a 40K-node IBM Blue Gene/P supercomputer and a 128-node Linux cluster. The traces span a 17-month period, cover 173M tasks, and have an average task runtime of 95 seconds. We propose methods to verify the existence of BOT arrival pattern, and ways to measure their impacts on system performance. We also examine the correlations among several BOT attributes, such as BOT size, runtime, CPU times, and inter-arrival time of BOT. The results show that the interarrival time of the two BOT workloads has Generalized Pareto (GP) distribution, and there are autocorrelations and cross-correlations among the BOT attributes.