An analysis of feature-based and state-based representations for module-based learning in mobile robots

The information available to robots in real tasks is widely distributed both in time and space, requiring the agent to search for relevant information. In this paper, we implement a solution that uses qualitative and quantitative knowledge to turn robot tasks able to be treated by reinforcement learning (RL) algorithms. The steps of this procedure include: 1) to decompose the overall task into smaller ones, using abstraction and macro-operators, thus achieving a discrete action space; 2) to use observation functions of the environment - here called features - to achieve both time and state space discretisation; 3) to use quantitative knowledge to design controllers that are able to solve the subtasks; 4) to learn the coordination of these behaviours using RL, more specifically Q-learning. The approach was verified on an increasingly complex set of robot tasks using a Khepera robot simulator. Two approaches for space discretisation were used, one based on features and the other on states. The learned policies over these two models were compared to a predefined hand-crafted one. It was found that the learned policy over the state-based discretisation leads quickly to good results, although it can not be applied to complex tasks, where the state space representation becomes computationally unfeasible.