A Dynamic MapReduce Scheduler for Heterogeneous Workloads

Author

Tian, Chao ; Zhou, Haojie ; He, Yongqiang ; Zha, Li

Author_Institution

Inst. of Comput. Technol., Chinese Acad. of Sci., Beijing, China

fYear

2009

Firstpage

218

Lastpage

224

Abstract

MapReduce is an important programming model for building data centers containing ten of thousands of nodes. In a practical data center of that scale, it is a common case that I/O-bound jobs and CPU-bound jobs, which demand different resources, run simultaneously in the same cluster. In the MapReduce framework, parallelization of these two kinds of job has not been concerned. In this paper, we give a new view of the MapReduce model, and classify the MapReduce workloads into three categories based on their CPU and I/O utilization. With workload classification, we design a new dynamic MapReduce workload predict mechanism, MR-Predict, which detects the workload type on the fly. We propose a Triple-Queue Scheduler based on the MR-Predict mechanism. The Triple-Queue scheduler could improve the usage of both CPU and disk I/O resources under heterogeneous workloads. And it could improve the Hadoop throughput by about 30% under heterogeneous workloads.

Keywords

data handling; resource allocation; scheduling; CPU utilization; CPU-bound job; Hadoop throughput; I/O utilization; I/O-bound job; MR-Predict mechanism; Triple-Queue Scheduler; data center; dynamic MapReduce scheduler; heterogeneous workload; resource usage; workload classification; Chaos; Computers; Dynamic programming; Dynamic scheduling; Grid computing; Hardware; Helium; Processor scheduling; Throughput; Web and internet services; MapReduce; Schdule; heterogeneous workloads;

fLanguage

English

Publisher

ieee

Conference_Titel

Grid and Cooperative Computing, 2009. GCC '09. Eighth International Conference on

Conference_Location

Lanzhou, Gansu

Print_ISBN

978-0-7695-3766-5

Type

conf

DOI

10.1109/GCC.2009.19

Filename

5279616