Benchmarking a virtualization platform

Traditional benchmarking of new architectures often involves running a workload on a single system with a single OS. In such a setup, the objective is typically to stress a single resource (e.g., CPU) and produce a single number used to characterize the performance of the system. Newer benchmarks have extended this paradigm by testing the performance of distributed systems like Hadoop clusters or cloud-style workloads such as big data analytics. These benchmarks are invaluable for evaluating physical systems, but have numerous drawbacks and gaps when used to evaluate virtualized systems. A virtualized system must be evaluated in an end-to-end way beyond what is done for physical systems, with measurements and workloads for the hypervisor platform, the application, and the management layer that supports virtualization-related services like live migration or rapid VM provisioning. In this paper, we describe an end-to-end approach to evaluating virtualization platforms. We begin with a discussion of traditional virtualization benchmarking, involving comparing performance on real systems vs. performance on virtualized systems. We next describe three benchmarks that we have developed specifically for virtualized environments. We discuss a system-level benchmark, VMmark, which incorporates workloads that simultaneously stress a number of different resources (CPU, memory, and IO). We then outline two additional benchmark suites designed for measuring virtualization management performance and application performance: VCBench and ViewPlanner. For each workload, we describe how it was developed, what it tests, and the key insights it has provided in terms of optimizing virtualized platforms. We conclude with a discussion of next-generation virtualization benchmarks.