Dynamic resource management in a MapReduce-style platform for fast data processing

There is a recent interest in building MapReduce-style platforms for fast data processing, such as MapReduce online [2] and Muppet [5]. In this paper, we highlight the need for dynamic load management in a distributed data stream processing system and present Enorm, a MapReduce-style data stream processing platform with the focus on techniques to achieve dynamic resource management, i.e. the ability to dynamically balance the workload among the running instances and scale the resource usage according to the runtime workload fluctuations. The original MapReduce framework is designed for batched processing and dynamic scaling can only be achieved between batches. To address this problem, we propose a MapReduce-style computation framework and a set of corresponding adaptation strategies that can perform dynamic scaling on the fly with low processing latency.