Supporting Multi-user Participation with Affective Multimodal Fusion

In this paper, we present an application of affective computing as an art installation designed for group interaction. The Common Touch utilises a large multi-touch display, presenting interactive visualisations of emotive slogans. The artistic brief is to engage participants in the exploration, touching and manipulation of slogans. Participants reveal the missing words of slogans by touching them. The Common Touch utilises several input modalities to build an affective representation of the group interactions: emotional speech recognition, video feature extraction, multi-keyword spotting and touch events. The output of affective fusion is used to refine the selection of slogans presented. We include results of a series of experiments using The Common Touch with 24 subjects in groups of 3 using video analysis, logs and questionnaires for data collection. We describe, through interaction analysis, how users utilised the different modalities, suggesting implications for implementing multimodal aesthetic applications to support multi-user participation: tracking multi-user engagement, coverage of possible affective cues in each modality, multimodality in temporal and event analysis, dramaturgy and performative interaction.