Efficient Value Function Approximation with Unsupervised Hierarchical Categorization for a Reinforcement Learning Agent

We investigate the problem of reinforcement learning (RL) in a challenging object-oriented environment, where the functional diversity of objects is high, and the agent must learn quickly by generalizing its experience to novel situations. We present a novel two-layer architecture, which can achieve efficient learning of value function for such environments. The algorithm is implemented by integrating an unsupervised, hierarchical clustering component into the Soar cognitive architecture. Our system coherently incorporates several principles in machine learning and knowledge representation including: dimension reduction, competitive learning, hierarchical representation and sparse coding. We also explore the types of prior domain knowledge that can be used to regulate learning based on the characteristics of environment. The system is empirically evaluated in an artificial domain consisting of interacting objects with diverse functional properties and multiple functional roles. The results demonstrate that the flexibility of hierarchical representation naturally integrates with our novel value function approximation scheme and together they can significantly improve the speed of RL.