Towards Economic Fairness for Big Data Processing in Pay-as-You-Go Cloud Computing

Recent trends indicate that the pay-as-you-go Infrastructure-as-a-Service (IaaS) cloud computing has become a popular platform for big data processing applications, due to its merits of accessibility, elasticity and flexibility. However, the resource demands of processing workloads are often varying over time for individual users, implying that it is hard for a user to keep the high resource utilization for cost efficiency all the time. Resource sharing is a classic and effective approach to improve the resource utilization via consolidating multiple users´ workloads. However, we show that, current existing fair policies such as max-min fairness, widely adopted and implemented in many popular big data processing systems including YARN, Spark, Mesos, and Dryad, are not suitable for pay-as-you-go cloud computing. We show that it is because of their memory less allocation feature which can arise a series of problems in the pay-as-you-go cloud environment, namely, cost-inefficient workload submission, untruthfulness and resource-as-you-pay unfairness. This paper presents these problems and outlines our plans to address them for pay-as-you-go cloud computing. We introduce our preliminary work done on the single-resource fairness and our ongoing work for multi-resource fairness, and outline our future work.