An active mesh based tracker for feature correspondences

We address the problem of obtaining feature correspondences from a set of image frames that comprise a video sequence. These correspondences are to be used subsequently within a 3-d estimation stage, based for example on the Fundamental Matrix or Trifocal Tensor. When working with video sequences, two major problems present themselves. Firstly adjacent video frames correspond to negligible camera motion leading to instability of the 3-d metric. The solution to this would be track features until the camera motion provides stability. This however leads directly to the second problem. Traditionally feature trackers are based upon finding the optic flow of an image patch located around an image feature. The problem is that image features usually correspond to say a junction of objects or the junction of object and background. The assumption that the composition of that patch remains constant is obviously violated under the 3-d motion that we seek to estimate. Our solution is to treat the problem as one of fitting an active mesh to the sequence. The optic flow of each mesh element then votes for the overall motion of its corner points to achieve the best fit of the mesh to each subsequent frame. We find that this method allows us to fit 3-d metrics to a high degree of accuracy.