A multimodal interface to situated assembly robot systems

In the following, we present the development of and experiment with a robot system showing multimodal interaction capabilities. We focus on the understanding of human instructions in natural language by integrating gaze and pointing hand gestures. A typical application of such a system is interactive assembly. A human communicator sharing a view of the assembly scenario with the robot instructs the latter by speaking to it in the same way that he would communicate with a child. His instructions can be under-specified, incomplete and/or context-dependent. After introducing the general purpose of our project, we present the hardware and software components of our system necessary for interactive assembly tasks. Finally, we outline a list of future research topics for extending our research.