Seeing in three dimensions: correlation and triangulation of Mars Exploration Rover imagery

The Mars Exploration Rovers (MER) uses a man-in-the-loop system for control in most cases. While capable of some autonomous driving, all arm operations and most drives are planned on the ground. Planning these operations requires a precise knowledge of the terrain surrounding of the rover: where are the rocks, the sand, and the hazards. This terrain is derived from images taken by stereoscopic cameras. This paper describes in detail the middle parts of the ground-based terrain derivation process: correlation, which finds matching points in the stereo pair, and triangulation, which converts those points to XYZ coordinates. The algorithms and free parameters are described, followed by a discussion of the results obtained, the problems encountered, and possible avenues for future development.