State space self-organization based on the interaction between basis functions

In an application of reinforcement learning to real-world problems, the function approximators are usually used to approximate the value function and the policy function. It is necessary to construct the function approximator adaptively, because the value function and the policy function change along with the progress of reinforcement learning. In this work, we propose self-organizing basis network (SOBN) which is the method that constructs a function approximator using basis functions adaptively. The proposed method constructs a basis function network by connecting neighbor bases using edges. This basis function network constrains the activating region of each basis function, and the network is modified by updating the location of each basis. Using this mutual dependence, which we call the interaction between basis functions, for searching appropriate architecture of a function approximator, SOBN self-organizes the function approximator. Assuming that the method is applied to reinforcement learning, we apply the method to the function approximation problem, and evaluate approximation performance and convergence time.