An Evaluation Approach for Dynamics-Aware Applications Using Linked Data

One possible threat to linked data quality is the lack of knowledge about the dynamics in dependent remote datasets. Linked data consuming applications often need to be aware of changes in these datasets in order to update local data dependencies. Dataset dynamics recently emerged as an active research topic and methods for the detection, propagation, and description of the dynamics in linked datasets are being developed. It is, however, difficult to compare these methods on a quantitative and qualitative level due to the lack of appropriate benchmarks and evaluation infrastructures. Therefore we have implemented a reusable infrastructure for the evaluation of applications that are concerned with linked dataset dynamics. Our contributions comprise a vocabulary for the description of timely-ordered changes in linked datasets and an extensible tool-set for extracting and simulating these changes. We also provide two example datasets and explain their usage in the evaluation of our own tool dealing with dynamic datasets. We consider these contributions as a first step towards a reference benchmark for measuring the performance of solutions dealing with issues resulting from dataset dynamics, such as low and high-level change detection or efficient and scalable notification mechanisms.