Safety critical neural computing: explanation and verification in knowledge augmented neural networks

Conventional neural networks cannot contain a priori knowledge and cannot explain their output. A mathematical theory of black box classifiers is developed which covers most of the best known neural architectures. The limitations of the non-model based computational paradigm are discussed; these include inability to predict the behaviour of systems with multiple-valued, discontinuous, catastrophic and chaotic state spaces. Worse, they include the inability to detect the presence of such systems, when they are working: outside their `range of competence´, and with data of quality outside their range of experience. Neural networks themselves cannot communicate with human decisionmakers in human terms; often the choice is `take it or leave it´. Knowledge-based computation does not necessarily have these drawbacks, and can therefore augment the powerful neural computing paradigm where it is weakest. The authors consider three fundamental ways of combining the two computational paradigms and show how the explanation facility of knowledge based systems can be used to induce explanation on the output of neural subsystems. They conclude with an architecture which is generic for safety critical neural computation