Generating Memoryless Policies Faster Using Automatic Temporal Abstractions for Reinforcement Learning with Hidden State

Reinforcement learning with eligibility traces has been an effective way to solve problems with hidden state. Under certain conditions, it succeeds to build up a memoryless optimal policy over observations. Automatic generation of temporal abstractions, on the other hand, provides ways to extract and make use of useful sub-policies during reinforcement learning for a fully observable problem setting, so that the agent shall not need to repeatedly learn the same skill. One of the recent automatic abstraction techniques is the extended sequence tree method. We propose a novel way to bring together the extended sequence tree method and reinforcement learning for problems with hidden state. We expand the extended sequence tree method with a mechanism that helps the abstraction procedure to get rid of adverse effects of perceptual aliasing, letting the agent to make use of the remaining useful abstractions. Effectiveness of the method is shown empirically via experimentation on some benchmark problems.