DocumentCode :
2388868
Title :
A two-phase execution engine of reduce tasks in Hadoop MapReduce
Author :
Xiaohong Zhang ; Guowei Wang ; Zijing Yang ; Yang Ding
Author_Institution :
Sch. of Comput. Sci. & Technol., Henan Polytech. Univ., Jiaozuo, China
fYear :
2012
fDate :
19-20 May 2012
Firstpage :
858
Lastpage :
864
Abstract :
In Hadoop MapReduce, reduce tasks issue massive remote I/O operations to copy the intermediate results of map tasks. The operations cause massive remote data access delays which degrade the system performance. To Handle this problem, we propose an execution engine of reduce tasks. The engine partitions the execution of reduce tasks into two phases. In the first phase, the engine selects the nodes to run reduce tasks, and then order the nodes to prefetch intermediate results for the reduce tasks. In the second phase, the selected nodes allocate computing and memory resource to the reduce tasks, and execute these tasks. Due to the fact that intermediate results have been prefetched, reduce tasks can access these results from local nodes, and the remote access delay of the results can be hidden. We have implemented the engine in Hadoop-0.20.2. We evaluated the engine in a Linux cluster. The results showed that the engine optimized the performance of Hadoop in most cases.
Keywords :
parallel processing; public domain software; resource allocation; software performance evaluation; storage allocation; storage management; task analysis; Hadoop MapReduce; Hadoop-0.20.2; Linux cluster; computing resource allocation; map tasks; massive remote data access delays; memory resource allocation; performance optimization; prefetching; reduce task execution; remote I/O operations; system performance degradation; two-phase execution engine; Data communication; Delay; Educational institutions; Engines; Prefetching; Servers; System performance; Data access Delay; Data intensive applications; MapReduce; Scheduling;
fLanguage :
English
Publisher :
ieee
Conference_Titel :
Systems and Informatics (ICSAI), 2012 International Conference on
Conference_Location :
Yantai
Print_ISBN :
978-1-4673-0198-5
Type :
conf
DOI :
10.1109/ICSAI.2012.6223144
Filename :
6223144
Link To Document :
بازگشت