A projector-camera system for creating a display with water drops

Various non-traditional media, such as water drops, mist, and fire, have been used to create vibrant two and three dimensional displays. Usually such displays require a great deal of design and engineering. In this work, we show a computer vision based approach to easily calibrate and learn the properties of a three-dimensional water drop display, using a few pieces of off-the-shelf hardware. Our setup consists of a camera, projector, laser plane, and water drop generator. Based on the geometric calibration between the hardware, a user can “paint” the drops from the point of view of the camera, causing the projector to illuminate them with the correct color at the correct time. We first demonstrate an algorithm for the case where no drop occludes another from the point of view of either camera or projector. If there is no occlusion, the system can be trained once, and the projector plays a precomputed movie. We then show our work toward a display with real rain. In real time, our system tracks and predicts the future location of hundreds of drops per second, then projects rays to hit or miss each drop.