Detecting social context: A method for social event classification using naturalistic multimodal data

As intelligent interactive systems become prevalent in human social environments, it is important they are able to understand social context. Humans process information to derive expected behaviors from their social context; however to date few autonomous systems have taken advantage of this rich resource of information. Social context processing is a broad problem area, and in this paper we focus specifically on social event detection. We used machine learning techniques to automatically classify 320 multinational YouTube videos of eight complex social events, including interviews, parties, weddings, and sporting events. This paper presents three major contributions. First, we demonstrate fairly high classification accuracy on extremely noisy, real-world data. Second, we show that a multimodal approach plays a significant role in achieving such accuracy (video and audio features together outperform either one alone). Third, we provide evidence that high level social information can be extracted from video automatically. These findings will be useful to researchers in the fields of affective computing, human-machine interaction, and robotics, as they design systems to leverage social context.