Title :
Do You Feel the Lag of Your Hadoop?
Author :
Yuxuan Jiang ; Zhe Huang ; Tsang, Danny H. K.
Author_Institution :
Dept. of Electron. & Comput. Eng., Hong Kong Univ. of Sci. & Technol., Hong Kong, China
fDate :
March 30 2015-April 2 2015
Abstract :
The configuration of a Hadoop cluster is significantly important to its performance, because an improper configuration can greatly deteriorate the job execution performance. Unfortunately, systematic guidelines on how to configure a Hadoop cluster are still missing. In this paper, we undertake an empirical study on key operations and mechanisms of Hadoop job execution, including the task assignment strategy and speculative execution. Based on the experiments, we provide suggestions on the system configuration, particularly on the matching between the hardware resource partitioning scheme and the job splitting granularity.
Keywords :
data handling; parallel processing; pattern clustering; resource allocation; Hadoop cluster configuration; Hadoop job execution; hardware resource partitioning scheme; job splitting granularity; task assignment strategy; Cloud computing; Delays; Electronic publishing; Hardware; Indexes; Resource management; Time-domain analysis; Empirical Study; Hadoop; Performance Measurement;
Conference_Titel :
Big Data Computing Service and Applications (BigDataService), 2015 IEEE First International Conference on
Conference_Location :
Redwood City, CA
DOI :
10.1109/BigDataService.2015.14
