Evaluating expert systems using a multiple-criteria, multiple-stakeholder approach

Test results of an approach to evaluating the quality of expert systems are presented. The solution proposed is a multiple-criteria- and multiple-stakeholder-based subjective assessment technique. Its validity was established by conducting field tests with operational expert systems at 19 sites in the North American insurance industry. The results indicate that such a socio-technical approach to evaluating expert systems is a promising alternative to current ad hoc and anecdotal practices. It is the long-term goal of such a line of investigation to produce a diagnostic tool that would serve to support feedback in the knowledge engineering life cycle