Benchmarking Eventual Consistency: Lessons Learned from Long-Term Experimental Studies

Cloud storage services and NoSQL systems typically guarantee only Eventual Consistency. Knowing the degree of inconsistency increases transparency and comparability, it also eases application development. As every change to the system implementation, configuration, and deployment may affect the consistency guarantees of a storage system, long-term experiments are necessary to analyze how consistency behavior evolves over time. Building on our original publication on consistency benchmarking, we describe extensions to our benchmarking approach and report the surprising development of consistency behavior in Amazon S3 over the last two years. Based on our findings, we argue that consistency behavior should be monitored continuously and that deployment decisions should be reconsidered periodically. For this purpose, we propose a new method called Indirect Consistency Monitoring which allows to track all application-relevant changes in consistency behavior in a much more cost-efficient way compared to continuously running consistency benchmarks.