Evaluating the effectiveness of the Rainbow self-adaptive system

Rainbow is a framework for engineering a system with run-time, self-adaptive capabilities to monitor, detect, decide, and act on opportunities for system improvement. We applied Rainbow to a system, Znn.com, and evaluated its effectiveness to self-adapt on three levels: its effectiveness to maintain quality attribute in the face of changing conditions, run-time overheads of adaptation, and the engineering effort to use it to add self-adaptive capabilities to Znn.com. We make Znn.com and the associated evaluation tools available to the community so that other researchers can use it to evaluate their own systems and the community can compare different systems. In this paper, we report on our evaluation experience, reflect on some principles for benchmarking self-adaptive systems, and discuss the suitability of our evaluation tools for this purpose.