Risk management: the irremedial incompleteness of rules

Risk management poses a major challenge in the design and operation of large-scale industrial systems. The common strategy for managing risk is the rule-based approach in which automatic safety devices and standard operating procedures are developed as a counter to the potentially catastrophic effects of system malfunctions that can be anticipated. This is, however, a flawed approach. Large-scale industrial systems can be so complex that it is impossible to anticipate all eventualities. In addition, rules are irremedially incomplete as determinants of behavior. They encapsulate explicit knowledge but they cannot be used to completely specify behavior because their use always involves implicit or unarticulated knowledge. A case study of a highly reliable system with catastrophic potential is described. The development of experiential knowledge is seen as a key element in the achievement of safe and reliable procedures. The case study suggests that operators should be given direct access to critical information about the process to be controlled and should be provided with ample opportunity to develop sensitivity to this critical information