Design and implementation of a multi-modal user interface of the Virtual World Database system (VWDB)

Reports on the design and implementation of a multi-modal user interface of the Virtual World Database (VWDB) system. VWDB is currently under development at Ochanomizu University with the aim of realizing a new-generation database system for cyberspace applications. The system is being constructed by integrating a virtual reality system and an object-oriented database system. Since VWDB provides a virtual world as the user´s database interface, the system must support a multi-modal database language with which users can interact with VWDB by means of gestures and voice. We introduce an automata-based approach to designing the multi-modal language. This idea is in response to the observation that an object in the virtual world changes its state according to a user´s query or an update request issued in a multi-modal manner. We identify a set of automata to characterize the multi-modal interaction and describe a semantic network approach for translating a multi-modal user input to an equivalent message to access and manipulate database objects.