DocumentCode :
3657142
Title :
Self-Configuration of the Number of Concurrently Running MapReduce Jobs in a Hadoop Cluster
Author :
Bo Zhang;Filip Krikava;Romain Rouvoy;Lionel Seinturier
Author_Institution :
INRIA, Univ. of Lille 1, Lille, France
fYear :
2015
fDate :
7/1/2015 12:00:00 AM
Firstpage :
149
Lastpage :
150
Abstract :
There is a trade-off between the number of concurrently running MapReduce jobs and their corresponding map and reduce tasks within a node in a Hadoop cluster. Leaving this trade-off statically configured to a single value can significantly reduce job response times leaving only sub optimal resource usage. To overcome this problem, we propose a feedback control loop based approach that dynamically adjusts the Hadoop resource manager configuration based on the current state of the cluster. The preliminary assessment based on workloads synthesized from real-world traces shows that the system performance can be improved by about 30% compared to default Hadoop setup.
Keywords :
"Time factors","Random access memory","Yarn","Feedback control","Analytical models","Containers","Memory management"
Publisher :
ieee
Conference_Titel :
Autonomic Computing (ICAC), 2015 IEEE International Conference on
Type :
conf
DOI :
10.1109/ICAC.2015.54
Filename :
7266952
Link To Document :
بازگشت