DOCTOR: an integrated software fault injection environment for distributed real-time systems

The paper presents an integrated software fault injection environment (DOCTOR) which is capable of (1) generating synthetic workloads under which system dependability is evaluated, (2) injecting various types of faults with different options, and (3) collecting performance and dependability data. A comprehensive graphical user interface is also provided. The software implemented fault-injection tools supports three types of faults: memory faults, CPU faults, and communication faults. Each injected fault may be permanent, transient or intermittent. A fault-injection plan can be formulated probabilistically, or based on the past event history. The modular organization of tools is particularly designed for distributed architectures. DOCTOR is implemented on a distributed real-time system called HARTS, and it capability has been tested through numerous experiments