A Symmetric Multiprocessor Architecture for Multi-Agent Temporal Difference Learning

Temporal difference learning methods have been successfully applied to a wide range of stochastic learning and control problems. In addition to correctness, one metric of a technique´s performance is its learning rate - the number of iterations required to converge to an optimal solution. The learning rate can be increased by using multiple agents that can share experience. In a software environment, the potential speedup from additional agents is limited, since adding agents significantly increases the burden of computation and/or hinders real-time processing. To address this problem, this paper presents a parameterized hardware model of a multi-agent system based on a shared-memory Symmetric Multiprocessor (SMP). To the author´s knowledge, this is the first application of an SMP architecture to a multi-agent reinforcement learning system. The control model employed is a multi-agent variation of the Sarsa(¿) algorithm. Several hardware optimizations schemes are investigated with respect to feasibility and expected performance. The system is modeled using a cycle-accurate simulation in SystemC. The results indicate that real-time learning rates can be significantly improved by employing the proposed parallel hardware implementation.