Trading off perception with internal state: reinforcement learning and analysis of Q-Elman networks in a Markovian task

A Markovian reinforcement learning task can be dealt with by learning a direct mapping from states to actions or values, or from state-action pairs to values. However, this may involve a difficult pattern recognition problem when the state space is large. This paper shows that using internal state, called “supportive state”, may alleviate this problem presenting an argument against the tendency to almost automatically use a direct mapping when the task is Markovian. This point is demonstrated in simulation experiments of an agent controlled by a neural network capable of learning the strategy of direct mapping as well as internal state, combining Q(λ) learning and recurrent neural networks in a new way. The trade-off between the two strategies is investigated in more detail, focusing particularly on border cases