Annotation of spoken syntax in relation to prosody and multimodal pragmatics

The HuComTech project at the University of Debrecen aims at studying and better understanding the multimodal properties of human-human communication both in order to build more user-friendly human-machine interaction systems and to assist a more successful communication between humans as well. A 60 hour multimodal corpus of two types of dialogues serves as the basis for a database of video, audio and pragmatic annotations. Most recently, challenging work has been ongoing involving the syntactic annotation of speech corpora and its relation to both multimodal and unimodal pragmatic events. This paper focuses on the theoretical background and some of the available results of this work.