3D structure from a monocular sequence of images

The authors address the following problem: given a camera moving in an unknown environment, they want to obtain a 3-D description of the environment. A unifying approach is presented by deriving a unique formalism to process uniformly different but complementary features, namely points and linear segments. Different concepts for tracking features are given: (1) 2-D tracker-2-D features are tracked using an order one dynamic model for their evolution; (2) 2-D+estimation tracker-3-D fusion of 2-D features is performed recursively, and then the value predicted at time t for these 3-D features is projected at time t+1 onto the camera focal plane and replaces the dynamic model used in the 2-D tracker, allowing the introduction of 3-D information into the 2-D feature tracker without prior knowledge of the environment; and (3) 3-D tracker-the 2-D tracker disappears, and all computations are 3-D. The 3-D tracker combines the simplicity of the 2-D tracker and the efficiency of the 2-D+estimation tracker. A description is given of the mechanisms of fusion that integrate 2-D measurements into an estimate of the feature 3-D parameters. Uncertainties are taken into account through extended Kalman filtering. Feature parametrizations are chosen to simplify the linearization process and ensure numerical stability