Real-time composition of stereo images for video see-through augmented reality

This paper describes a method of stereo image composition for video see-through augmented reality. In order to implement an augmented reality system, we must acquire the position and orientation of the user´s viewpoint to display the composed image maintaining correct registration of real and virtual worlds. All the procedures must be done in real-time. We have built a prototype augmented reality system that adopts the combination of a vision-based tracking technique and a video see-through head mounted display (HMD). Display-timing is synchronized between the real and virtual environments, so that an alignment error is reduced. The system calculates camera parameters from three markers among which physical relationships are unknown in image sequences captured by a pair of stereo cameras mounted on the HMD. In addition, the user´s hands are regarded as real-world objects that may occlude virtual objects; the system estimates the depth of hands in images and generates a composed image maintaining consistent occlusions between the hands and virtual objects