Loosely-Coupled Benchmark Framework Automates Performance Modeling on IaaS Clouds

Cloud computing is under rapid development, which brings the urgent need of evaluation and comparison of cloud systems. The performance testers often struggle in tedious manual operations, when carrying out lots of similar experiments on the cloud systems. However, few of current benchmark tools provide both flexible workflow controlling methodology and extensible workload abstraction at the same time. We present a modeling methodology to compare the performance from multiple aspects based on a loosely coupled benchmark framework, which automates experiments under agile workflow controlling and achieves broad cloud supports, as well as good workload extensibility. With several built-in workloads and scenario templates, we performed a series of tests on Amazon EC2 services and our private Open Stack-based cloud, and analyze the elasticity and scalability based on the performance models. Experiments show the robustness and compatibility of our framework, which makes remarkable guarantee that it can be leveraged in practice for researchers and testers to perform their further study.