Title :
Dynamic data rebalancing in Hadoop
Author :
Ashwin Kumar, T.K. ; Jongyeop Kim ; George, K.M. ; Park, Nahea
Author_Institution :
Dept. of Comput. Sci., Oklahoma State Univ., Stillwater, OK, USA
Abstract :
Current implementation of Hadoop is based on an assumption that all the nodes in a Hadoop cluster are homogenous. Data in a Hadoop cluster is split into blocks and are replicated based on the replication factor. Service time for jobs that accesses data stored in Hadoop considerably increases when the number of jobs is greater than the number of copies of data and when the nodes in Hadoop cluster differ much in their processing capabilities. This paper addresses dynamic data rebalancing in a heterogeneous Hadoop cluster. Data rebalancing is done by replicating data dynamically with minimum data movement cost based on the number of incoming parallel mapreduce jobs. Our experiments indicate that as a result of dynamic data rebalancing service time of mapreduce jobs were reduced by over 30% and resource utilization is increased by over 50% when compared against Hadoop.
Keywords :
parallel processing; pattern clustering; replicated databases; resource allocation; Hadoop cluster data; data access; data replication; dynamic data rebalancing service time; heterogeneous Hadoop cluster; jobs service time; minimum data movement cost; parallel mapreduce jobs; replication factor; resource utilization; Clustering algorithms; Dynamic scheduling; Equations; Heuristic algorithms; Load management; Mathematical model; Resource management; Dynamic Data Rebalancing; Hadoop; Replication; heterogeneity; service time; waiting time;
Conference_Titel :
Computer and Information Science (ICIS), 2014 IEEE/ACIS 13th International Conference on
Conference_Location :
Taiyuan
DOI :
10.1109/ICIS.2014.6912153
