Adaptive Fuzzy Function Approximation for Multi-agent Reinforcement Learning

Reinforcement learning has difficulties in solving multi-agent problems because of the inefficiency of function approximation. Sparse distributed memories, which is implemented using Radial Basis Functions or Kanerva Coding, can be used to improve the efficiency. But this approach still often give poor performance when applied to large-scale multi-agent systems. In this paper, we attempt to solve a collection of instances in the predator-prey pursuit domain and argue that the poor performance that we observe is caused by frequent prototype collisions. We show that dynamic prototype allocation and adaptation can give better results by reducing these collisions. We then describe our novel approach, fuzzy Kanerva-based function approximation, that uses a fine-grained fuzzy membership grade to describe a state-action pair´s adjacency with respect to each prototype. This approach completely eliminates prototype collisions. We further show that prototype density varies widely across the state-action space and that this variation causes prototypes´ receptive fields to be unevenly distributed. This distribution limits the ability of fuzzy Kanerva Coding to achieve better results. We demonstrate that another advantage of fuzzy Kanerva Coding is that it allows prototypes to tune their receptive fields for a target application. We conclude that fuzzy Kanerva Coding with prototype tuning and adaptation can significantly improve a reinforcement learner´s ability to solve large-scale multi-agent problems.