Referring to objects with spoken and haptic modalities

The gesture input modality considered in multimodal dialogue systems is mainly reduced to pointing or manipulating actions. With an approach based on spontaneous character of the communication, the treatment of such actions involves many processes. Without constraints, the user may use gesture in association with speech, and may exploit visual context peculiarities, guiding her/his articulation of gesture trajectories and her/his choice of words. Semantic interpretation of multimodal utterances also becomes a complex problem, taking into account varieties of referring expressions, varieties of gestural trajectories, structural parameters from the visual context, and also directives from a specific task. Following the spontaneous approach, we propose to give maximal understanding capabilities to dialogue systems, to ensure that various interaction modes must be taken into account. Considering the development of haptic sense devices (such as PHANToM) which increase the capabilities of sensations, particularly tactile and kinesthetic, we propose to explore a new domain of research concerning the integration of haptic gesture into multimodal dialogue systems, in terms of its possible associations with speech for object reference and manipulation. We focus on the compatibility between haptic gesture and multimodal reference models, and on the consequences of processing this new modality on intelligent system architectures, which has been sufficiently studied from a semantic point of view.