Scheduling Divisible Loads from Multiple Input Sources in MapReduce

MapReduce is a programming model for data-intensive applications. The performance of MapReduce is greatly affected by the scheduling efficiency, which needs to consider both the parallelism of processing and overhead of data transmission. Lots of researches only focus on optimizing scheduling mechanisms or data storage strategies to reduce the overhead of data transmission in big data applications. The analysis of MapReduce carried out doesn´t consider the overhead of data transmission sufficiently. This paper proposes a divisible load scheduling model to describe the data transmission and task execution in MapReduce. The task allocation and corresponding data transmission are abstracted as scheduling divisible loads from multiple input sources. With the established model, the optimal distribution of input data and scheduling of loads in map phase are presented with linear programming. The performance of map phase is evaluated and analyzed under different environments.