Parallel Workload Modeling with Realistic Characteristics

Workload modeling and performance evaluation play crucial roles in the study of scheduling algorithms on large-scale parallel and distributed systems. An effective design of a scheduling algorithm for these systems requires experiments with hundreds of simulations to evaluate its performance. Since each simulation needs one workload as input, only real workloads with usually a limited availability are not sufficient, and so representative workload models are needed. Several studies have shown that realistic workload characteristics such as burstiness, bag-of-tasks, etc., cause significant performance impacts on scheduling. Therefore, we argue that realistic workload models should contain as many characteristics of real workloads as possible. In practice, researchers use unrealistic workloads in their scheduling evaluations because they lack models that can help generate realistic workloads. In this article, we analyze real parallel workloads to show the presence of important characteristics including long range dependence, periodicity and temporal burstiness of job arrivals, bag-of-tasks behavior, and correlation of runtime and number of processors. Then, we present a systematic approach to create a complete model that contains all of these characteristics. Validation of our model with real world data shows that it does not only capture the above characteristics, but also can fit marginal distributions well.