Cooperative Behavior Acquisition Based Modular Q Learning in Multi-Agent System

In a multi-agent system, action selection is important for the cooperation and coordination among agents. The better cooperative behaviors can usually be acquired by reinforcement learning. The overlap of actions selected by each agent makes the acquisition of cooperative behaviors less efficient. To solve the problem we propose a novel learning architecture. The architecture consists of several learning modules which amount to the number of agents involved in the task, a generalized rule base module and a final action selection module. Every learning module includes a Q learning module and an ASPL (Action Selection Priority Level) module. ASPL module to determine the action selection priority level based on which the cooperative behaviors can be well controlled. Generalized rule base module stores the environment states, Q value and action parameters etc. information, and extracts action rules based Q value from learning process. Final action selection module decides agent’s final action based on ASPL and generalized rule base. The learning architecture can decrease environment state space and agent’s action space, and accelerate learning speed of agents. We have applied the proposed method to a robot soccer match and the efficiency was verified by the results of both the computer simulation and real experiments.