Acquisition and use of transferable, spatio-temporal plan representations for human-robot interaction

Service robots that work together with humans in domestic and constantly changing environments should have a general understanding about their human partners and the tasks that are to be performed. This would enable them to verify their beliefs about the common tasks and the goals of their human partners and detect unexpected events and failures. In this paper we present a way of acquiring general, spatio-temporal plan representations from human motion tracking data in different environments. Using an annotated data set for table setting tasks in a typical kitchen environment, we first cluster the static positions of the participants and create a spatial model relative to furniture objects that are given by a semantic map and linked to a knowledge base. Based on this spatial model learned in one kitchen, we automatically generate spatio-temporal plan representations in different kitchen environments with known semantic maps. We show that our models can successfully be used to give a robot a basic understanding about a task executed by a human in three different environments. We evaluate the quality of our automatic generation of the plan representations and present an example application of plan supervision using a learned model from one kitchen to differentiate tasks performed by humans in other kitchens.