Grey-Box Approach for Performance Prediction in Map-Reduce Based Platforms

Map-Reduce has become an important paradigm for data-intensive computations. The ability to estimate Map-reduce application performance is critical for efficient resource scheduling and provisioning both on dedicated clusters and on the cloud. Current state-of-the-art techniques for performance prediction of Map- Reduce applications use analytical and simulation-based models. In this paper, we make the case for performance prediction using regression techniques based on machine-learning. Through modeling the Map-Reduce environment as a grey-box, we can leverage a combination of externally observed system features and information about sub-system internals. We identify four learning techniques with high prediction accuracy through a detailed comparative study of twenty methods. The powerful capabilities of data analytics platforms are usually accompanied by frequent faults that occur due to scale, complexity and the use of commercial off- the-shelf components. We show that our proposed approach can effectively predict degraded performance under these faulty conditions by the inclusion of additional fault-related input features. A mean prediction error of <;12% was achieved across the range of parameters studied on a 64-node Xen virtualized environment running an open-source Map-Reduce implementation, Hadoop.