Conversational multi-modal browser: an integrated multi-modal browser and dialog manager

This paper describes a multi-modal, free flow browser. A multi-modal browser allows the user to perform tasks via any of several modalities, such as speech and GUI or devices available sequentially or concurrently. A free-flow or mixed initiative browser allows the user to choose the navigation across an application and the order in which tasks are completed. This paper describes a system to provide flow conversational capability across modalities or devices. Multi-modal interaction is provided using a DOM based MVC multi-modal browser extended with an event-driven form-based dialog manager. We use XForms-based device independent authoring for the application. This paper details the XForms-based programming model of conversational applications, the principles and architecture of an XForms-based and event driven, task oriented, mixed initiative dialog manager and the integration of a dialog manager and a multi-modal browser using a standard-based DOM event driven interface. The dialog manager relies on a DOM based representation of the form-based framework presented by Papineni (1999) and Davies et al. (1999) to provide free-flow dialog management. It determines the user´s intention by measuring the relevance of user input to different tasks. It also provides focus determination by determining the next logical turn in the conversation.