Title :
Scheduling Hadoop Jobs to Meet Deadlines
Author :
Kc, Kamal ; Anyanwu, Kemafor
Author_Institution :
Dept. of Comput. Sci., North Carolina State Univ., Raleigh, NC, USA
fDate :
Nov. 30 2010-Dec. 3 2010
Abstract :
User constraints such as deadlines are important requirements that are not considered by existing cloud-based data processing environments such as Hadoop. In the current implementation, jobs are scheduled in FIFO order by default with options for other priority based schedulers. In this paper, we extend real time cluster scheduling approach to account for the two-phase computation style of MapReduce. We develop criteria for scheduling jobs based on user specified deadline constraints and discuss our implementation and preliminary evaluation of a Deadline Constraint Scheduler for Hadoop which ensures that only jobs whose deadlines can be met are scheduled for execution.
Keywords :
data handling; parallel processing; pattern clustering; public domain software; scheduling; FIFO order; Hadoop job scheduling; MapReduce; cloud based data processing environment; deadline constraint scheduler; priority based scheduler; real time cluster scheduling approach; two phase computation style; user constraint; Data models; Data processing; Estimation; Processor scheduling; Real time systems; Resource management; Runtime;
Conference_Titel :
Cloud Computing Technology and Science (CloudCom), 2010 IEEE Second International Conference on
Conference_Location :
Indianapolis, IN
Print_ISBN :
978-1-4244-9405-7
Electronic_ISBN :
978-0-7695-4302-4
DOI :
10.1109/CloudCom.2010.97
