Fusing 2D and 3D clues for 3D tracking using visual and range data

3D tracking of rigid objects is required in many applications, such as robotics or augmented reality (AR). The availability of accurate pose estimates increases reliability in robotic applications and decreases jitter in AR scenarios. Pure vision-based 3D trackers require either manual initializations or offline training stages, whereas trackers relying on pure depth sensors are not suitable for AR applications. In this paper, an automated 3D tracking algorithm, which is based on fusion of vision and depth sensors via Extended Kalman Filter (EKF), which inherits a novel observation weighting method, is proposed. Moreover, novel feature selection and tracking schemes based on intensity and shape index map (SIM) data of 3D point cloud, increases 2D and 3D tracking performance significantly. The proposed method requires neither manual initialization of pose nor offline training, while enabling highly accurate 3D tracking. The accuracy of the proposed method is tested against a number of conventional techniques and superior performance is observed.