Robust view-based visual tracking with detection of occlusions

Visual tracking plays an important role in various robotic tasks. We have developed a view-based visual tracking system to cope with change of the appearance of the template in a 3D environment using affine transformed templates. In view-based visual tracking, however, there are still critical situations in which the tracking fails because of occlusion by general objects and/or human hand in the environment, particularly in cases where humans and robots perform cooperative handling of objects. The paper presents two methods to cope with such occlusion problems. The first method detects the occlusion based on an evaluation of correlation errors in a tessellated template. The second method uses infrared images to detect the region occluded by human hand in the target template. The system then creates a mask of the occluded region and eliminates pixels in the mask from the calculation of correlation with the template image. In the prototype system, the generation of the mask and the correlation can be performed, typically, in one TV frame. We have integrated the proposed methods into the view-based visual tracking system which we have already developed. Experimental results demonstrate the usefulness of the proposed methods.