Towards benchmarks for knowledge systems and their implications for data engineering

The author suggests a new focus on benchmarks for knowledge systems, following the lines of similar benchmarks in other computing fields. It is noted that knowledge systems differ from conventional systems in a key way, namely their ability to interpret and apply knowledge. This gives rise to a distinction between intrinsic measures concerned with engineering qualities and extrinsic measures relating to task productivity, and both warrant improved measurement techniques. Primary concerns within the extrinsic realm include advice quality, reasoning correctness, robustness, and solution efficiency. Intrinsic concerns, on the other hand, center on elegance of knowledge base design, modularity, and architecture. The author suggests criteria for good measures and benchmarks, and ways to satisfy these through the design of knowledge and key knowledge engineering costs and performance parameters. It is suggest that the focus on measuring knowledge systems should help clarify the technical relationships between knowledge engineering and data engineering