Controlling generic visualization environments using handheld devices and natural feature tracking

This paper presents a framework that enables human-machine interaction with generic visualization devices using personal consumer electronics equipment and scaleinvariant feature matching image processing techniques. The proposed implementation tracks the location and orientation of a generic mobile device equipped with an onboard camera and makes it possible to develop innovative and affordable user interfaces. The framework matches the time-variant features extracted from the frame buffer of a 2D or 3D scene with the features extracted from the camera-generated images that contain the displayed scene in order to derive positioning information with six degrees of freedom. Experimental results regarding precision, accuracy and latency confirmed the applicability of this solution as both an alternative user interface for desktop applications as well as an affordable method to support interaction with virtual environments.