Feedback of robot states for object detection in natural language controlled robotic systems

Controlling robots with natural language enables untrained users to interact with them more easily. A significant challenge for such systems is the mismatched visual perceptual capabilities between humans and robots. Most existing methods try to improve the perceptual ability of robots by either developing robust vision algorithms to describe and identify objects more accurately, or refining the object segmentation through human collaboration. In this paper, we present a novel method to detect and track objects, and even discover previously undetected objects (e.g. objects occluded by or stacked on other objects) by incorporating feedback of robot states into the vision module. By reasoning about the object states according to the trajectories of robot states and then re-detecting the point clouds of the objects, the representation of the environment can be efficiently and accurately updated. Experimental results demonstrate the effectiveness and advantages of the proposed method.