Requirements for the benchmarking of decentralized manufacturing control systems

The research community of manufacturing control systems has a high demand for reference systems to evaluate and compare their algorithms. Most of the reference systems used in scientific works are not fully described in publications or even undefined, making valid comparisons of different algorithms impossible. In this article, we evaluate the state-of-the-art in reference systems and derive requirements for benchmarking such systems to increase comparability. Based on these requirements, we define three Complexity Dimensions: operational scenario, plant scenario, and transport system scenario. We utilize these Complexity Dimensions to generate a set of 36 scenarios to be used in reference systems for making them more suitable for comparison and evaluation of manufacturing control algorithms.