On the optimization of Hadoop MapReduce default job scheduling through dynamic job prioritization

One of the most popular frameworks for big data pro- cessing is Apache Hadoop MapReduce. The default Hadoop scheduler uses queue system. However, it does not consider any specic priority for the jobs required for MapReduce programming model. In this paper, a new dynamic score is developed to improve the per- formance of the default Hadoop MapReduce scheduler. This dynamic priority score is computed based on eec- tive factors such as job runtime estimation, input data size, waiting time, and length or bustle of the waiting queue. The implementation of the proposed schedul- ing method, based on this dynamic score, not only im- proves CPU and memory performance, but also reduced waiting time and average turnaround time by approxi- mately 45% and 40% respectively, compared to the de- fault Hadoop scheduler.