Measurement issues in knowledge engineering

There are five sources (or determinants) of knowledge base quality: domain experts, knowledge engineers, knowledge representation schemes, knowledge elicitation methods, and problem domains. The knowledge base for many expert systems is developed for a problem domain using one domain expert, one knowledge engineer, one knowledge representation scheme, and one elicitation method. Since there is minimal research demonstrating that the possible variation in each of these sources does not significantly affect the quality of the knowledge base, the generalizability (or validity) of such systems in real-world settings is questionable. Consequently, research is needed to assess the extent to which system validity is affected by these sources of variability. Toward this end, the results of reanalyzing the data from an experiment varying domain experts, knowledge engineers, and elicitation methods when developing a multiattributed representation scheme for combat readiness are presented. No significant effects were obtained for elicitation method or knowledge engineer