Do You Feel the Lag of Your Hadoop?

Author

Yuxuan Jiang ; Zhe Huang ; Tsang, Danny H. K.

Author_Institution

Dept. of Electron. & Comput. Eng., Hong Kong Univ. of Sci. & Technol., Hong Kong, China

fYear

2015

fDate

March 30 2015-April 2 2015

Firstpage

115

Lastpage

119

Abstract

The configuration of a Hadoop cluster is significantly important to its performance, because an improper configuration can greatly deteriorate the job execution performance. Unfortunately, systematic guidelines on how to configure a Hadoop cluster are still missing. In this paper, we undertake an empirical study on key operations and mechanisms of Hadoop job execution, including the task assignment strategy and speculative execution. Based on the experiments, we provide suggestions on the system configuration, particularly on the matching between the hardware resource partitioning scheme and the job splitting granularity.

Keywords

data handling; parallel processing; pattern clustering; resource allocation; Hadoop cluster configuration; Hadoop job execution; hardware resource partitioning scheme; job splitting granularity; task assignment strategy; Cloud computing; Delays; Electronic publishing; Hardware; Indexes; Resource management; Time-domain analysis; Empirical Study; Hadoop; Performance Measurement;

fLanguage

English

Publisher

ieee

Conference_Titel

Big Data Computing Service and Applications (BigDataService), 2015 IEEE First International Conference on

Conference_Location

Redwood City, CA

Type

conf

DOI

10.1109/BigDataService.2015.14

Filename

7184871