Fuzzy C-means method for representation policy iteration in reinforcement learning

This paper introduces a Fuzzy C-means method as the subsampling method for Representation Policy Iteration (RPI) in Reinforcement Learning. RPI is a new class of algorithm that automatically learns both basis functions and approximately optimal policy. In this paper the procedures of the RPI algorithm are as follows. Firstly samples are collected using a random or guided policy. The subset samples are obtained from the original ones subsequently by using the Fuzzy C-means (FCM) method as the subsampling method. Then global basis functions called proto-value functions (PVFs) are formed by using the eigenfunctions of the graph Laplacian operator on an undirected graph constructed from the subset samples. Finally, the least-square policy iteration (LSPI) as the parameter estimation method is used for learning an approximately optimal policy. Illustrative experiments on an Inverted Pendulum problem were accomplished to compare the performance of RPI using the FCM method as the subsampling method with that using the previous subsampling method.