Clustering Similar Actions in Sequential Decision Processes

The presence of a large number of available actions in the context of an automated, adaptive decision process can lead to an excessively large search space and thus significantly increase the overhead for the policy learning process. This issue occurs particularly in problem domains such as path planning or grid scheduling where the number of decision points is large and irreducible. The learning algorithm developed in this paper attempts to create a more compact representation of the state and action space by grouping similar actions that are likely leading to very similar future results. Actions are considered similar if they, with high probability, lead to future results with sufficient commonality. This paper develops this action clustering framework within the MDP formalism where actions in any given state are grouped if they result in similar reinforcement feedback based on the past learning experience. The resulting action sets are then considered as a whole in the decision process.