Integration of work operation and embodied multimodal interaction in task modeling for collaborative robot development

In a collaborative robot development, the robot is required to understand both the content of the work operation, and the current condition of the working environment and the progress of the human, in order to provide exact assistance in the correct timing. This work aims to integrate together both robot knowledge in work operation and embodied multimodal interaction for collaborative robot development. A task modeling approach is proposed to build a hierarchical task model of the entire work operation and grounding with the actual condition (“state”) of the objects and environment, and the action (“transition”) of work in the embodied dimension. The system is implemented in a simulation environment with human interaction in both structured (collaborative cooking task) and unstructured (clean up task) operations.