Title :
A novel run-time load balancing method for MapReduce
Author :
Zhihong Liu; Yaping Liu; Baosheng Wang; Zhenghu Gong
Author_Institution :
College of Computer, National University of Defense Technology, Changsha, Hunan, China
Abstract :
In recent years, many companies are embracing the Hadoop MapReduce system for large-data processing with completion time constrains. However, exiting Hadoop schedulers still suffer from the reducer load imbalancing problem. In this paper, we present a novel run-time load balancing method for MapReduce. Our approach predicts the workload of each reduce task at run-time, and assigns the reduce tasks to specified machines based on the estimated workload of reduce tasks dynamically. Therefore, our approach can achieve load balance among machines. The experimental results show that our approach achieves high accuracy while predicting the workload of reduce tasks, and improves the job completion time by up to 23.15%.
Keywords :
"Load management","Load modeling","Monitoring","Biomedical monitoring","Training data","Approximation algorithms","Companies"
Conference_Titel :
Computer Science and Network Technology (ICCSNT), 2015 4th International Conference on
DOI :
10.1109/ICCSNT.2015.7490726