Dealing with Query Contention Issue in Real-Time Data Warehouses by Dynamic Multi-level Caches

The issue of query contention and scalability is the most difficult issue facing organizations deploying real-time data warehouse solutions. The contention between complex selects and continuous inserts tends to severely limit the scalability of the data warehouses. In this paper, we present a new method called dynamic multi-level caches, to effectively deal with the problem of query contention and scalability in real-time data warehouses. We differentiate between queries with various data freshness requirements, and use multi-level caches to satisfy these different requirements. Every query arriving at the system will be automatically redirected to the corresponding cache to access the required data, which means that the query loads are distributed to multi-level caches instead of becoming blocked in the only one cache due to the contention between query and update operations. Extensive experiments on several real datasets show that our method can effectively balance the query loads among multi-level caches and achieve desirable system performance.