Smart Task Distributor for MapReduce on Cloud Computing

A MapReduce system is widely used to implement the large-scale computation on cloud computing. A MapReduce system currently defines computation resources in a node as a roughly configurable slot number, and distributes tasks over nodes according to the slot number. However, a MapReduce system may make computation resources of clusters in the underutilization or overutilization condition, because a task of different applications unlikely uses the same computation resources and because a node may have different CPUs with different capabilities. A MapReduce system can use Smart Task Distributor (STD) proposed in this paper to solve the problem of the computation resource underutilization or overutilization in clusters. Technically, a MapReduce system can use STD to smartly distribute tasks over nodes in clusters, because STD on the one hand gradually assigns tasks to a node in order to fully utilize computation resources in the node and on the other hand dynamically estimates the remaining computation resources in the node for toggling the assignment of tasks on demand without overloading it. In experiments, a MapReduce system is proved to get better performances with STD than with other ways.