VrtProf: Vertical Profiling for System Virtualization

As data centers and end users become increasingly reliant on virtualization technology, more efficient and accurate methods of profiling such systems are needed. However, under virtualization the virtual machine and OS each try to manage the same resources independently, the underlying hardware is now multiplexed between many streams of execution, and non-trivial interference can be caused by seemingly unrelated resources. While sampling techniques are effective at gathering average behaviors over long runs, understanding the time-varying behavior of programs under virtualization, the correlation between events at the level of program phases, or the transient effects of rare events requires a new way of profiling virtualized applications. To this end we present VrtProf, a low overhead profiling tool that automates the collection of hardware and software events spanning the vertical execution stack, including the hardware, the virtual machine monitor, the guest kernels, and applications at very fine time scales. We describe the many challenges faced while developing VrtProf, the design of the resulting tool, and how it can be used in practice on several multi-programmed workloads both as virtualized and native executions. We show that VrtProf introduces negligible performance overhead of only 1.2% while capturing the time-varying application behavior with interval granularities as small as a few thousand cycles.