FRESH: Fair and Efficient Slot Configuration and Scheduling for Hadoop Clusters

Author

Jiayin Wang ; Yi Yao ; Ying Mao ; Bo Sheng ; Ningfang Mi

Author_Institution

Dept. of Comput. Sci., Univ. of Massachusetts Boston, Boston, MA, USA

fYear

2014

fDate

June 27 2014-July 2 2014

Firstpage

761

Lastpage

768

Abstract

Hadoop is an emerging framework for parallel big data processing. While becoming popular, Hadoop is too complex for regular users to fully understand all the system parameters and tune them appropriately. Especially when processing a batch of jobs, default Hadoop setting may cause inefficient resource utilization and unnecessarily prolong the execution time. This paper considers an extremely important setting of slot configuration which by default is fixed and static. We proposed an enhanced Hadoop system called FRESH which can derive the best slot setting, dynamically configure slots, and appropriately assign tasks to the available slots. The experimental results show that when serving a batch of MapReduce jobs, FRESH significantly improves the makespan as well as the fairness among jobs.

Keywords

Big Data; parallel processing; scheduling; FRESH; Hadoop clusters; Hadoop setting; MapReduce jobs; enhanced Hadoop system; parallel big data processing; resource utilization; scheduling; slot configuration; system parameter; Clustering algorithms; Dynamic scheduling; Heuristic algorithms; Indexes; Measurement; Optimized production technology; Resource management; MapReduce; Resource Management; Scheduling;

fLanguage

English

Publisher

ieee

Conference_Titel

Cloud Computing (CLOUD), 2014 IEEE 7th International Conference on

Conference_Location

Anchorage, AK

Print_ISBN

978-1-4799-5062-1

Type

conf

DOI

10.1109/CLOUD.2014.106

Filename

6973812