Manipulation of multiple objects in close proximity based on visual hierarchical relationships

This paper presents a method for daily assistive robots to manipulate objects whose contact relationships are important. In our approach, contact relationships between multiple objects such as “on” and “into” are estimated using the hierarchical relationships between the objects´ regions in the images. The hierarchical states are used for failure recovery processes so that the target contact state is achieved. We tested our method in two daily tasks by HRP-2VZ humanoid robot: filing, and serving an egg with a spatula. These results suggested the effectiveness of our approach.