A hand-held augmented reality projection system using trifocal tensors and Kalman filter

In this paper, we proposed a tensor-based hand-held projection system that makes use of pure computer vision techniques. Using a projector and two cameras, the system is able to spatially augment content on movable rigid surfaces, such as a map or a book. The advantage of our system is that neither attachment of sensors to the projection surface nor the prior reconstruction of the surface structure is required. Application examples include projecting information on a map to provide travel plan guidance or additional information on interesting sites such as hospitals, banks etc. To achieve this, a calibrated stereo camera pair is used as the tracking system for the projected surface and a non-calibrated projector is used for the projection. Our main contribution is on using trifocal tensors to lock the positions of the cameras, the projection surface and the projector. Augmented content is then generated and projected to the required surface accurately. Experimental results show that our system can project the augmented content onto the projection surface in high precision in real-time.