Smart Intermediate Data Transfer for MapReduce on Cloud Computing

MapReduce is a programming model proposed by Google to process large datasets in clusters. However, MapReduce often needs to transfer much intermediate data among nodes, which is harmful to performances of an application. MapReduce can be enhanced by using the proposed Smart Intermediate Data Transfer (SIDT) in the runtime system to smartly arrange intermediate data. Although SIDT does not reduce intermediate data to the minimal size in comparison with other intermediate data arrangement procedures such as Huffman coding, bzip2, and gzip, MapReduce is proved to get a better performance from SIDT than from others in the experiments of this paper.