Title :
CooMR: Cross-task coordination for efficient data management in MapReduce programs
Author :
Xiaobing Li ; Yandong Wang ; Yizheng Jiao ; Cong Xu ; Yu Weikuan
Author_Institution :
Dept. of Comput. Sci. & Software Eng., Auburn Univ., Auburn, AL, USA
Abstract :
Hadoop is a widely adopted open source implementation of MapReduce programming model for big data processing. It represents system resources as available map and reduce slots and assigns them to various tasks. This execution model gives little regard to the need of cross-task coordination on the use of shared system resources on a compute node, which results in task interference. In addition, the existing Hadoop merge algorithm can cause excessive I/O. In this study, we undertake an effort to address both issues. Accordingly, we have designed a cross-task coordination framework called CooMR for efficient data management in MapReduce programs. CooMR consists of three component schemes including cross-task opportunistic memory sharing and log-structured I/O consolidation, which are designed to facilitate task coordination, and the key-based in-situ merge (KISM) algorithm which is designed to enable the sorting/merging of Hadoop intermediate data without actually moving the <;key, value> pairs. Our evaluation demonstrates that CooMR is able to increase task coordination, improve system resource utilization, and significantly speed up the execution time of MapReduce programs.
Keywords :
Big Data; data handling; distributed programming; public domain software; sorting; Big Data processing; CooMR; Hadoop intermediate data; Hadoop merge algorithm; KISM algorithm; MapReduce programming model; compute node; cross-task coordination framework; cross-task opportunistic memory sharing; data management; excessive I/O; key-based in-situ merge algorithm; log-structured I/O consolidation; shared system resources; system resource utilization; task interference; Abstracts; Arrays; Software engineering;
Conference_Titel :
High Performance Computing, Networking, Storage and Analysis (SC), 2013 International Conference for
Conference_Location :
Denver, CO
Print_ISBN :
978-1-4503-2378-9