A projector-camera system with real-time photometric adaptation for dynamic environments

Projection systems can be used to implement augmented reality, as well as to create both displays and interfaces on ordinary surfaces. Ordinary surfaces have varying reflectance, color, and geometry. These variations can be accounted for by integrating a camera into the projection system and applying methods from computer vision. The methods currently applied are fundamentally limited since they assume the camera, projector, and scene are static. In this paper, we describe a technique for photometrically adaptive projection that makes it possible to handle a dynamic environment. We begin by presenting a co-axial projector-camera system whose geometric correspondence is independent of changes in the environment. To handle photometric changes, our method uses the errors between the desired and measured appearance of the projected image. A key novel aspect of our algorithm is that we combine a physics-based model with dynamic feedback to achieve real time adaptation to the changing environment. We verify our algorithm through a wide variety of experiments. We show that it is accurate and runs in real-time. Our algorithm can be applied broadly to assist HCI, visualization, shape recovery, and entertainment applications.