Combined Wireless Hardware and Real-Time Computer Vision Interface for Tangible Mixed Reality

Recent advances in technology enable portable, even wearable, computers to be equipped with wireless interfaces, which allows data transactions even while mobile. Combined with mixed reality (MR), mobile computing exploits a promising field for wearable computers. Natural and nonobtrusive means of interaction call for new devices, which should be simple to use, and provide effective tracking methods in unprepared environments for MR. In this paper, a new interaction hardware tilt pad designed using accelerometers and wireless devices is introduced. This is combined with two new natural feature-tracking algorithms based on geometrical image constraints. The first is based on epipolar geometry and provides a general description of the constraints on image flow between two static scenes. The second is based on the calculation of a homography relationship between the current frame and a stored representation of the scene. We assessed these algorithms compared with the current optical flow calculation algorithm across a number of criteria including robustness, speed, and accuracy. Finally, we demonstrated an MR computer game application combining the new tracking method and the hardware tilt pad. Videos of the tilt pad and application of tilt pad Pacman game can be found at the web site: http://www.mixedrealitylab.org.