An Efficient Multimodal Language Processor for Parallel Input Strings in Multimodal Input Fusion

Multimodal user interaction technology aims at building more natural and intuitive interfaces allowing a user to interact with a computer in a way similar to human-to-human communication, for example, through speech and gesture. As a critical component in multimodal user interaction, multimodal input fusion explores the ways to effectively interpret the combined semantic interpretation of user inputs through multiple modalities. This paper proposes a new efficient unification-based multimodal language processor which can handle parallel input strings for multimodal input fusion. With a structure sharing technology, it has the potential to achieve a low polynomial computational complexity while parsing multimodal inputs in versatile styles. The applicability of the proposed processor has been validated through an experiment with multimodal commands collected from traffic incident management scenarios. The description of the proposed multimodal language processor and preliminary experiment results are presented.