Human-robot teamwork using activity recognition and human instruction

We address the problem of robot participation as a team member in group activities. In this human-robot interactive setting, the goal is to enable a robot to detect objects around it using on-board video cameras, and determine which activity they are performing. It then determines how it should move to participate in the activity based on its intended role, derived from domain knowledge, and human instruction. A human controller can initialize, correct and refine the robot´s behavior and recognition of surrounding activities. A probabilistic framework is used for joint estimation of the robot´s desired motion and the activity surrounding it while incorporating human instruction. Group activities consisting of multiple people moving about the robot with a common goal are used to demonstrate the proposed approach. We focus on activities with geometric structures (line, wedge, echelon, etc.) with an arbitrary number of participants, and handle the noisy observations from video detection and tracking. Results using both simulated and real video data are used to demonstrate activity recognition and incorporating human instruction.