A usage-aware scheduler for improving MapReduce performance in heterogeneous environments

Big data cannot be efficiently dealt with using most relational database management systems, as usually it requires parallel execution on a large amount of servers. MapReduce is suitable for processing large data sets, however, most traditional MapReduce schedulers assume that system is homogeneous and all tasks are executed equally in time. In reality, the completion time of a MapReduce job may be delayed due to slower tasks. This paper presents a usage-aware MapReduce scheduler to deal with the system heterogeneity by including task execution time in scheduling. Inspiration from the ideas of both the Fair scheduler and LATE scheduler, our usage-aware scheduler is able to reduce the overall completion time of MapReduce applications. Experimental results show that a reduction of up to 30% of job execution time is attainable.