Automated and Isolated Tests for Complex Middleware Products: The Case of BPEL Engines

Today, a plethora of enterprise middleware solutions are available, leading to the problem of choosing the right tool for a specific use case. Automated tests can support the selection of such software by determining decision relevant metrics, like e.g., throughput or the degree of standard conformance. To avoid side effects between tests, test isolation, i.e., to provide fresh instances of the software for each test execution, is essential. However, middleware suites are inherently complex, provide a large range of configuration options, have tedious or sometimes manual installation procedures, and long startup times. These idiosyncrasies aggravate the creation of fresh instances of such middleware suites, leading to slower turnaround times and increasing the cost for ensuring test isolation. We aim to overcome these issues with methods and tools from the area of virtualization and devops. In this work, we focus on BPEL engines which are common middleware components in Web Service based SOAs. We applied our proposed method to the BPEL Engine Test System (betsy), a conformance test suite and testing tool for BPEL engines. Results reveal that our method a) enables automatic creation of fresh instances of software without manual installation steps, b) reduces the time to create these fresh instance dramatically, and c) introduces only a neglectable performance overhead, therefore, reducing the overall costs of testing complex software.