IMU-compensated image segmentation for improved vision-based control performance

Traditionally, image segmentation algorithms are viewed as black-box measurement sources. In this work, the system-level formulation of closed-loop tracking is reformulated under the assumption of access to physical sensor data in addition to the video stream. The scope is restricted to algorithms that iteratively minimize a functional to localize the target in each image frame. A nominal iterative algorithm is an open-loop system; IMU data is then used to generate a control to modify the algorithm´s behavior and thus create a closed-loop system. Compensating using IMU data in a feedback setting creates the possibility of induced track loss, whereby the compensation effectively undoes the action of the motion control. This issue is overcome by solving an optimization program that exploits known control intent to separate ego-motion disturbances from the control. As a result, the reformulation proposed in this paper enables improved tracking of a fast-moving target even in the presence of strong disturbances/jitter in the camera platform.