Trust, systems and accidents: designing complex systems

Trust is easier to destroy than create, hence, trust in overly applied automation may erode following accidents. This paper looks at the crash of a revolutionary supersonic fighter that resulted from over-reliance on protection technology. The protection system has been automated to the extent that it was impossible for the pilot to regain control and convince the system that there was a problem. Complete trust in the safety of the system has thus been translated into a new kind of computer-assisted error, where in the interest of safety the only possible exit strategy is outlawed by the system leading to the potential (or in this case, the actual) destruction of the system it was meant to protect. Overall, trust appears to be an emergent function that takes in safety issues affecting the entire system as well as the relationship between the product, client and developer and the trade-offs that underpin decisions. Operators, working as part of a system, abrogate some of their responsibility by implicitly or explicitly passing on control to computerised systems. While operators function on as an integral part of systems, their trust in the ability and safety of the system plays a key role in the ensuing success or failure.