XConveryer: Guarantee Hadoop Throughput via Lightweight OS-Level Virtualization

Large-scale data parallel applications such as web indexing, data mining demand plenty of computing and storage resources. As a widely adopted solution, hadoop partitions and distributes the large datasets into chunks across multiple nodes in clusters to process in parallel. For a single cluster node, typically there are several concurrently running applications, sharing and competing system CPU, memory, disk and network bandwidth. The competition will cause unfairness for some jobs and extend their response time. Particularly, some jobs have to be cancelled and rescheduled because of resource starvation. In this paper, we present a solution to reduce resource challenge. The XConveyer, as one of key component, is designed based on kernel-level virtualization facilities, to isolate Hadoop jobs and to guarantee their resource share. Built on XConveyer, Hadoop can enforce scheduling polities down to Operating System, that more fine-granular control can be achieved. The evaluation shows the XConveyer can encapsulate Hadoop jobs executed in isolated containers and guarantee strict control their system resources usages.