On the analysis of experimental results in evolutionary computation

Evolutionary computation methods are successfully applied in solving of combinatorial optimization problems. Since the “No Free Lunch” theorem states that there is no single best algorithm to solve all possible problems, throughout the years many algorithms and their modifications have emerged. When a new algorithm is developed, one question that naturally arises is how it compares to other algorithms, whether for some specific problem or in general performance. Because of the stochastic nature of systems involved, usually the only possible way of deriving the answer is to perform extensive experimental analysis. In this paper we provide an overview of possible approaches in the experimental analysis, and describe statistical methods that could be used. Furthermore, we outline similarities and differences between these methods, which lead to a discussion of important issues that need to be resolved when using these methods.