Scheduling Hadoop Jobs to Meet Deadlines

Author

Kc, Kamal ; Anyanwu, Kemafor

Author_Institution

Dept. of Comput. Sci., North Carolina State Univ., Raleigh, NC, USA

fYear

2010

fDate

Nov. 30 2010-Dec. 3 2010

Firstpage

388

Lastpage

392

Abstract

User constraints such as deadlines are important requirements that are not considered by existing cloud-based data processing environments such as Hadoop. In the current implementation, jobs are scheduled in FIFO order by default with options for other priority based schedulers. In this paper, we extend real time cluster scheduling approach to account for the two-phase computation style of MapReduce. We develop criteria for scheduling jobs based on user specified deadline constraints and discuss our implementation and preliminary evaluation of a Deadline Constraint Scheduler for Hadoop which ensures that only jobs whose deadlines can be met are scheduled for execution.

Keywords

data handling; parallel processing; pattern clustering; public domain software; scheduling; FIFO order; Hadoop job scheduling; MapReduce; cloud based data processing environment; deadline constraint scheduler; priority based scheduler; real time cluster scheduling approach; two phase computation style; user constraint; Data models; Data processing; Estimation; Processor scheduling; Real time systems; Resource management; Runtime;

fLanguage

English

Publisher

ieee

Conference_Titel

Cloud Computing Technology and Science (CloudCom), 2010 IEEE Second International Conference on

Conference_Location

Indianapolis, IN

Print_ISBN

978-1-4244-9405-7

Electronic_ISBN

978-0-7695-4302-4

Type

conf

DOI

10.1109/CloudCom.2010.97

Filename

5708475