Research of an off-policy BF optimization aloorithm based on cross-entroropy method

In the reinforcement leaning task, the off-policy algorithms which approximately evaluate the values of states faced with the problem of high evaluation error and were sensitive to the distribution of behavior policy. In order to solve these problems, the basis function optimization method under the off-policy scenario was proposed. The algorithm set the Bellman error of the target policy which was computed with off-policy prediction algorithms as the objective function, then adjust the placement and shape of the basis functions in cooperate with the method of cross-entropy optimization. The experimental results on the grid world show that the algorithm effectively reduced the evaluation error and improved the approximation. Additionally, the algorithm could be easily extended to the problems of large state spaces.