RobusTest: A Framework for Automated Testing of Software Robustness

Robustness of a software system is defined as the degree to which the system can behave ordinarily and in conformance with the requirements in extraordinary situations. By increasing the robustness many failures which decrease the quality of the system can be avoided or masked. When it comes to specifying, testing and assessing software robustness in an efficient manner the methods and techniques are not mature yet. This paper presents RobusTest, a framework for testing robustness properties of a system with currently focus on timing issues. The expected robust behavior of the system is formulated as properties. The properties are then used to automatically generate robustness test cases and assess the results. An implementation of RobusTest in Java is presented here together with results from testing different, open-source implementations of the XMPP instant messaging protocol. By executing 400 test cases that were automatically generated from properties on two such implementations we found 11 critical failures and 15 nonconformance problems as compared to the XMPP specification.