Performance and Reliability Effects of Multi-tier Bidding on MapReduce in Auction-Based Clouds

Hadoop has become a central big data processing framework in today´s cloud environments. Ensuring the good performance and cost effectiveness of Hadoop is crucial for the numerous applications that rely on it. In this paper we analyze Hadoop´s performance in a multi-tier market-oriented cloud infrastructure known as Spot Instances. Amazon Spot Instances (SIs) are designed to deliver a cheap but transient alternative to fixed cost On-Demand (ODIs) instances. Recently, AWS introduced SIs in their managed Elastic Map Reduce offering. This managed framework lets the users design a multi-tier Hadoop architecture using fine grained controls to define the instance types both in terms of capacity, i.e. compute/storage/network, but also in terms of costs, i.e. ODI vs SI. The performance effects of such fine grained configurations are not yet well understood. First, we analyze a set of cluster configurations that can lead to important performance effects that can affect both the running time and the cost of such cloud Hadoop clusters. Second, we examine Hadoop´s fault tolerance mechanisms and show the inadequacy of these mechanisms for multi-tier bidding architectures. Third, we discuss directions for making the Hadoop framework more market-aware without losing its focus on extreme scalability.