DeltaDB: A Scalable Database Design for Time-Varying Schema-Free Data

DeltaDB is a model for a database consisting of records with no fixed schema whose entire history is captured over time. It is designed to support efficient queries against the current state of the database, any point in the history of the database, and historical data aggregations over time. In this paper, we present the DeltaDB data model, the associated query algebra, and highlight the fundamental query optimization concerns. To gain experience with the DeltaDB model, we have created a single-node implementation of the database and used it to collect one year´s worth of monitoring data from a distributed software system, reducing over 5TB of snapshots into 11GB of log data. We give examples of novel types of queries that exploit the time-varying nature of the data and evaluate their performance. We conclude with a discussion of how the single-node implementation will serve as the building block for a future distributed implementation.