3D pose tracking with linear depth and brightness constraints

This paper explores the direct motion estimation problem assuming that video-rate depth information is available, from either stereo cameras or other sensors. We use these depth measurements in the traditional linear brightness constraint equations, and we introduce a new depth constraint equation. As a result, estimation of certain types of motion, such as translation in depth and rotations out of the image plane, becomes more robust. We derive linear brightness and depth change constraint equations that govern the velocity field in 3-D for both perspective and orthographic camera projection models. These constraints are integrated jointly over image regions according to a rigid-body motion model, yielding a single linear system to robustly track 3D object pose. Results are shown for tracking the pose effaces in sequences of synthetic and real images