Multiple camera model-based 3-D visual servo

A multiple camera, visual servo system is presented that tracks and aligns rigid objects in 3D using wire-frame models. The system is capable of achieving frame-rate (30 Hz between two cameras) object tracking while using nonlinear iterative pose estimation. Tracking begins with a rough estimate of the positions of the objects. The relative rotation and translation of the cameras (external calibration) is calculated from the images. Six degree-of-freedom object pose estimates are computationally fused to provide tracking that is both stable and robust to occlusion. The tracking data is used to move (servo) the objects together while performing online robot Jacobian estimation. The robot is controlled in real time by joint angles with no prior knowledge of its kinematics