The added value of multimodality in the NESPOLE! speech-to-speech translation system: an experimental study

Multimodal interfaces, which combine two or more input modes (speech, pen, touch...), are expected to be more efficient, natural and usable than single-input interfaces. However, the advantage of multimodal input has only been ascertained in highly controlled experimental conditions (S.L. Oviatt, 1997; 1999); in particular, we lack data about what happens with "real" human-human, multilingual communication systems. We discuss the results of an experiment aiming to evaluate the added value of multimodality in a "true" speech-to-speech translation system, the NESPOLE! system, which provides for multilingual and multimodal communication in the tourism domain, allowing users to interact through the Internet sharing maps, Web-pages and pen-based gestures. We compared two experimental conditions differing as to whether multimodal resources were available: a speech-only condition (SO), and a multimodal condition (MM). Most of the data show tendencies for MM to be better than SO.