Continuous valued Q-learning method able to incrementally refine state space

The conventional reinforcement learning method has problems in applying to real robot tasks, because such method must be able to represent the values in terms of infinitely many states and action pairs. In order to represent an action value function continuously, a function approximation method is usually applied. In our previous work (2000), we pointed out that this type of learning method potentially has a discontinuity problem of optimal actions for a given state. In this paper, we propose a method for estimating where a discontinuity of the optimal action takes place and for refining a state space incrementally. We call this method an continuous valued Q-learning method. To show the validity of our method, we apply the method to a simulated robot