Performance of Left Outer Join on Hadoop with Right Side within Single Node Memory Size

In this paper we compare performance results of different implementations of join operation in Hadoop in a scenario where right side (of the join) is within single node memory size. We present results for several implementations both in pure Map Reduce and in Pig, both basing on HDFS. We also compare distributed performance of those implementations with a single node implementation in MySQL. Results show that Pig implementations do not match pure Map Reduce versions by a bigger margin than expected. Moreover, we notice that Map tasks seem to be the element that influences performance the most, especially for the potentially more efficient methods. Currently, we achieved the best performance using a singleton pattern join. However, there are reasons to believe that this performance can be still improved with better control of the amount of Map tasks.