DocumentCode :
692264
Title :
Distributed offline load balancing in MapReduce networks
Author :
Charalambous, Themistoklis ; Kalyvianaki, Evangelia ; Hadjicostis, Christoforos ; Johansson, Mikael
Author_Institution :
Sch. of Electr. Eng., R. Inst. of Technol. (KTH), Stockholm, Sweden
fYear :
2013
fDate :
10-13 Dec. 2013
Firstpage :
835
Lastpage :
840
Abstract :
In this paper we address the problem of balancing the processing load of MapReduce tasks running on heterogeneous clusters, i.e., clusters composed of nodes with different capacities and update cycles. We present a fully decentralized algorithm, based on ratio consensus, where each mapper decides the amount of workload data to handle for a single user job using only job specific local information, i.e., information that can be collected from directly connected neighboring mappers, regarding their current workload usage and capacity. In contrast to other algorithms in the literature, the proposed algorithm can be deployed in heterogeneous clusters and can operate asynchronously in both directed and undirected communication topologies. The performance of the proposed algorithm is demonstrated via simulation experiments on large-scale strongly connected topologies.
Keywords :
distributed processing; resource allocation; topology; MapReduce networks; directed communication topologies; distributed offline load balancing; fully decentralized algorithm; heterogeneous clusters; job specific local information; large-scale strongly connected topologies; processing load; ratio consensus; undirected communication topologies; update cycles; Clustering algorithms; Delays; Distributed algorithms; Educational institutions; Equations; Information exchange; Topology;
fLanguage :
English
Publisher :
ieee
Conference_Titel :
Decision and Control (CDC), 2013 IEEE 52nd Annual Conference on
Conference_Location :
Firenze
ISSN :
0743-1546
Print_ISBN :
978-1-4673-5714-2
Type :
conf
DOI :
10.1109/CDC.2013.6851767
Filename :
6851767
Link To Document :
بازگشت