An Introduction to Cloud Benchmarking

Over the last few years, more and more Cloud Computing offerings have emerged ranging from compute, data storage, and middleware services over platform environments up to ready-to-use applications. Choosing the best offering for a particular use case, is a complex task which involves comparison and trade-off analysis of functional and non-functional service properties; for non-functional quality of service (QoS) properties, this is typically done via benchmarking. Today, a plethora of benchmarking solutions exist for different layers in the cloud stack (IaaS, PaaS, SaaS) which typically address a single QoS dimension - a holistic cloud benchmark even for a single layer in the cloud stack is still missing. In this tutorial, we will give an overview of existing cloud benchmarking solutions and point-out ways in which these different benchmarks could be used in concert to actually compare clouds as a whole (i.e., for instance Amazon cloud vs. Google cloud) instead of analyzing isolated QoS dimensions of single cloud services.