Towards a Robustness Evaluation Framework for BPEL Engines

The selection of the best fitting process engine for a specific project requires the evaluation of engines according to various requirements. We focus on the non-functional requirement robustness, which is critical in production environments but hard to determine. Thus, we propose an evaluation framework to reveal important robustness criteria of process engines. In this work, we focus on message robustness, i.e., The ability to handle the receipt of invalid messages appropriately. In a case study comprising five open source BPEL engines, we determine message robustness by injecting faults into robustly designed processes as a reply to a previously sent request from an external virtual service and assert their behavior. The results show that the degree of message robustness significantly differs, hence, robustly designed processes do not necessarily lead to robust runtime behavior, the selected engines still play a major role.