Multimodal user interfaces in ubiquitous sensorised environments

The paper aims to show where ubiquitous sensor environments in combination with multimodal user interfaces can be used to enhance further the user (and usage) experience. The challenge in multimodal user interface design is to combine the available user input modalities with the available sensor data in a meaningful way. A conceptual framework for sensorised modality input composition and semantically controlled output decomposition in ubiquitous sensorised environments is introduced and its individual components, the ubiquitous sensorised environment, the input processing component, the situation aware application component and the output preparation component, are described. The paper also illustrates the functionality by means of an example, namely user mood awareness, and its application. Finally, considerable focus is put on the sensor nodes, classed into independent, parasitic and viral sensors, and some examples of what sensor payloads need to be employed to facilitate mood sensing are given.