An SDN App for Hadoop Clusters

The Apache Hadoop software library has been a widely used framework for extremely large-scale reliable, scalable, and distributed computing. Previous researches pointed out that the shuffle phase can be the performance bottleneck of a MapReduce job because of network congestion in the Hadoop data center. To address this issue, the idea of introducing the Software-Defined Networking (SDN) technology into Hadoop has been proposed in the literature. In this research, we facilitate the deployment of flow rules for Hadoop applications by implementing an SDN App for Hadoop clusters, and use the SDN App in a small-scale experimental Hadoop cluster to verify the effectiveness of using SDN to accelerate the execution of MapReduce jobs. To the best of our knowledge, this is the first SDN App targeted at Hadoop applications.