Model-Based Indirect Learning Method Based on Dyna-Q Architecture

In this paper, a model learning method based on tree structures is present to achieve the sample efficiency in stochastic environment. The proposed method is composed of Q-Learning algorithm to form a Dyna agent that can used to speed up learning. The Q-Learning is used to learn the policy, and the proposed method is for model learning. The model builds the environment model and simulates the virtual experience. The virtual experience can decrease the interaction between the agent and the environment and make the agent perform value iterations quickly. Thus, the proposed agent has additional experience for updating the policy. The simulation task, a mobile robot in a maze, is introduced to compare the methods, Q-Learning, Dyna-Q and the proposed method. The result of simulation confirms the proposed method that can achieve the goal of sample efficiency.