Video augmentation by image-based rendering under the perspective camera model

In this paper video streams are augmented by novel views of an object generated by an image-based rendering method. Reference views of the object and video images are captured by general uncalibrated perspective cameras. Our algorithm can be divided into three parts: 1) recovering structure information for reference views and video images; 2) interactive embedding of the model object into video images; and 3) rendering views of the object according to the motion of the video camera. Multiple model views are used to render views as the video camera changes its viewpoint. Generated views of the model are overlayed on the corresponding video images. Experimental results for real video and model images are given. Finally, a discussion on the limitations of the proposed method and subjects for future research are provided