Lumipen: Projection-Based Mixed Reality for Dynamic Objects

Recently, mixed reality (MR) involving the use of a projector, which is often called projection-based MR, has attracted much attention and is being widely studied. However, most MR research is mainly focused on static objects or situations. In contrast, we propose "Lumipen," which is a projection-based MR system for high-speed or high-frequency objects, using a high-speed vision sensor and a projector with a high-speed optical gaze controller. The high-speed vision sensor detects the dynamics of the object, and the high-speed optical gaze controller controls the projection direction in a few milliseconds. Lumipen is expected to enable addition of sophisticated and diverse visual information even in dynamic live content (e.g., a play, a sports game, a music concert, etc.), like computer graphics added to video content.