Incremental Refinement of Solutions for Dynamic Multi Objective Optimization Problems

MDQL is an algorithm, based on reinforcement learning, for solving multiple objective optimization problems, that has been tested on several applications with promising results. MDQL discretizes the decision variables into a set of states, each associated with actions to move agents to contiguous states. A group of agents explore this state space and are able to find Pareto sets applying a distributed reinforcement learning algorithm. The precision of the Pareto solutions depends on the chosen granularity of the states. A finer granularity on the states creates more precise solutions but at the expense of a larger search space, and consequently the need for more computational resources. In this paper, a very important improvement is introduced into the original MDQL algorithm to incrementally refined the Pareto solutions. The new algorithm, called IMDQL, starts with a coarse granularity to find an initial Pareto set. A vicinity for each of the Pareto solutions in refined and a new Pareto set is founded in this refined state space. This process continues until there is no more improvement within a small threshold value. It is shown that IMDQL not only improves the solutions found by MDQL, but also converges faster. MDQL has also been tested on the solutions of dynamic optimization problems. In this paper, it is also shown that the adaptation capabilities observed in MDQL can be improved with IMDQL. IMDQL was tested on the benchmark problems proposed by Jin. Performance evaluation was made using the Collective Mean Fitness metric proposed by Morrison. IMDQL was compared against an standard evolution strategy with the covariance matrix adaptation (CMA-ES) with very promising results.