State-Clusters shared cooperative multi-agent reinforcement learning

To hire multiple agents cooperating to solve realworld problem with large state space, a precondition is to provide an interaction medium for knowledge exchange and share among agents. We propose an interaction medium: State-Clusters, computed from the state trajectory that the agent wandered in state space. The State-Clusters of a state includes acyclic state paths from other states to this state, which represents the state space knowledge the agent learned. The State-Clusters brings two advantages: 1) it speeds up the convergence of value function, because the refined value function of a state can immediately propagate back to every states in its State-Clusters along the state path between them instead of requiring the agent wanders these state paths again; 2) it forms the substantial interaction medium with which agents can exchange and share state space knowledge with one another. Based on the State-Clusters, we extend Q-learning to multi-agent setting, to be a new cooperative multi-agent reinforcement learning approach. In this approach, each agent can use all State-Clusters produced by it and other agents to propagate refined value function to other states, even to these it never reached. This makes the value function converge faster, thus shorten the learning process. The experiments show this approach applied in two agents Q-learning outperform significantly single-agent Q-learning.