DocumentCode
2027913
Title
A Mapreduce programming framework using message passing
Author
Ho, Yu-Fan ; Chen, Sih-Wei ; Chen, Chang-Yi ; Hsu, Yung-Ching ; Liu, Pangfeng
Author_Institution
Dept. of Comput. Sci. & Inf. Eng., Nat. Taiwan Univ., Taipei, Taiwan
fYear
2010
fDate
16-18 Dec. 2010
Firstpage
883
Lastpage
888
Abstract
MapReduce is a very popular parallel programming model for processing large data sets. This paper discusses strategies in implementing a MapReduce runtime system using Message Passing Interface (MPI) library. The implementation uses blocking communication function in MPI, e.g. MPI_Send and MPI_Recv, to transfer intermediate data, so as to make the communication between mappers and reducers in MapReduce model much more efficient. Experiment results indicate that our MPI implementation performs better than Hadoop when the data volume is below 60MB, and perform five times better then native Hadoop when the input size is below 5MB.
Keywords
message passing; parallel programming; very large databases; MapReduce runtime system; Mapreduce programming; blocking communication function; data volume; large data sets; message passing interface library; parallel programming; Computational modeling; Computer architecture; File systems; Google; Message passing; Programming; Runtime; MPI; MapReduce;
fLanguage
English
Publisher
ieee
Conference_Titel
Computer Symposium (ICS), 2010 International
Conference_Location
Tainan
Print_ISBN
978-1-4244-7639-8
Type
conf
DOI
10.1109/COMPSYM.2010.5685386
Filename
5685386
Link To Document