On-line Object Reconstruction and Tracking for 3D Interaction

This paper presents a flexible and easy-to-use tracking method for 3D interaction. The method reconstructs points of a user-specified object from a video sequence, and recovers the 6 degrees of freedom (DOF) camera pose and position relative to the reconstructed points in each video frame. As opposed to most existing 3D object tracking methods, the proposed method does not need any off-line modeling or training process. Instead, it first segments the object from the background, then reconstructs and tracks the object using the Visual Simultaneous Localization And Mapping (VSLAM) techniques. To our knowledge, there are no existing works investigating this kind of on line reconstruction and tracking of moving objects. The proposed method employs the adapted pyramidal Lucas-Kanade tracker to increase the stability and the robustness of the tracking when dealing with a lightly textured or fast moving object. Experiments show that fast, accurate, stable and robust tracking can be achieved in everyday environment. Moreover, a simple stereo initialization approach is adopted to minimize user intervention. All these attributes conspire to make the method an adequate tool for some interaction applications. As a concrete example, an interactive 3D scene displaying system is demonstrated.