Local and Global Optimization of MapReduce Program Model

Author

Liu, Congchong ; Zhou, Shujia

Author_Institution

Univ. of Maryland, Baltimore, MD, USA

fYear

2011

fDate

4-9 July 2011

Firstpage

257

Lastpage

264

Abstract

MapReduce, which was introduced by Google, provides two functional interfaces, Map and Reduce, for a user to write the user-specific code to process the large amount of data. It has been widely deployed in cloud computing systems. The parallel tasks, data partition, and data transit are automatically managed by its runtime system. This paper proposes a solution to optimize the MapReduce program model and demonstrate it with X10. We develop an adaptive load distribution scheme to balance the load on each node and consequently reduce across-node communication cost occurring in the Reduce function. In addition, we exploit shared-memory in each node to further reduce the communication cost with multi-core programming.

Keywords

cloud computing; optimisation; parallel programming; resource allocation; shared memory systems; Google; MapReduce program model; X10; adaptive load distribution scheme; cloud computing system; data partition; data transit; global optimization; multicore programming; parallel task; shared memory system; user specific code; Adaptation models; Computational modeling; Computers; Distributed databases; Load modeling; Monitoring; Runtime; MapReduce; X10 parallel programming language; distributed memory system; load balancing; shared-memory system;

fLanguage

English

Publisher

ieee

Conference_Titel

Services (SERVICES), 2011 IEEE World Congress on

Conference_Location

Washington, DC

Print_ISBN

978-1-4577-0879-4

Electronic_ISBN

978-0-7695-4461-8

Type

conf

DOI

10.1109/SERVICES.2011.64

Filename

6012722