Expert systems for safety-critical applications: theory, technology and applications

In designing expert systems the problem of intrinsic safety is only one of the challenges facing application designers. Such systems use knowledge of an application domain in order to recommend, or possibly take, actions. As with conventional software it is necessary to ensure the integrity of the contents of the knowledge base and the decision making functions which use it. However, knowledge is always incomplete, often uncertain, or the limits on its applicability may be poorly understood. Consequently a knowledge base may have no demonstrable errors or inconsistencies in it, and the programs which apply it may operate as intended, yet the advice given by an expert system may be sub-optimal or even unsafe. This problem of consequential safety is an inevitable consequence of attempting to build practical technologies on the basis of some current state-of-the-art in any field which is not fully understood. This paper is concerned with issues concerning the design of such systems whose advice may have implications for safety. The RED project is attempting to address this difficulty by: exploring expert system applications which have safety implications and identifying general rules for safety reasoning; formalising a decision procedure in which decisions can explicitly apply these safety rules, and adapting formal methods for software engineering to rigorous specification and verification of knowledge bases