An Analysis and Hierarchical Decomposition for HAMs

In the HRL field, there are several main methods such as HAMs, options, MAXQ. These methods all rely on the theory of SMDPs. However, SMDPs does not specify how the overall task can be decomposed into a collection of subtasks. This paper introduces the concept of ldquopolicy-coupledrdquo SMDPs into HAMs. It defines the concept of HAM-decomposable and makes the relations among the HAM machine, HAM-decomposable, and ldquopolicy-coupledrdquo SMDPs clear. It also proves that HAMs is suitable for solving the ldquopolicy-coupledrdquo SMDPs problem. Based on these, this paper gives a method for hierarchical decomposition on a class of ldquopolicy-coupledrdquo SMDPs with a DAG call graph and presents a precondition that can be used for determining whether or not can generate a valid hierarchical decomposition. Lastly, a typical experiment is tested for illustrating the characteristics of this method.