Model-Based Design of Interactions That can Bridge Realities - The Augmented "Drag-and-Drop"

The development of Augmented Reality (AR) applications is still driven by development at source-code level. Although recent approaches focus to standardize AR functionality, to our knowledge a declarative and model-driven design (MDD) has not been applied for AR development so far. MDD approaches have been successfully applied to model user interfaces for a wide spectrum of modes (such as speech-command interfaces, or remote controls) and media such as HTML, 3D, and smart phone interfaces for instance. In this paper we propose a MDD approach for modeling seamless interaction between Web and AR interfaces. Therefore we implemented a prototype of a web furniture shop that supports arranging furniture in an augmented reality to prove our approach. We then present our approach of modeling a reality spanning Drag-and-Drop interaction between a 2D browser and an AR scene. Finally, we discuss the issues that we were confronted with to support reality spanning interaction like switching interaction modes and coordinate systems and present limitations that we have experienced with using other control modes, like a Wii-Controller or glove-based gesture detection.