How to Specify and Validate Dependability Benchmarks for Transactional Systems

Dependability benchmarking has been subject of intense research in recent years, having already led to the proposal of dependability benchmarks for database systems. However, the methodology to define dependability benchmarks is not clear yet. Furthermore, and more important, the validation of the proposed benchmarks has been disregarded so far. This question is particularly acute, as the ultimate goal of a dependability benchmark is to allow the comparison of different systems/components, which means that without proper validation of the benchmark one may draw wrong conclusions. This paper proposes a set of general guidelines for the definition and validation of dependability benchmarks for database centric On-Line Transaction Processing (OLTP) systems. The close relationship between the definition of a dependability benchmark and the validation aspects is also addressed.