EventMask: A Game-Based Framework for Event-Saliency Identification in Images

The concept of “event” emerged in recent years as a key feature to efficiently index and retrieve media. Several approaches have been proposed to analyze the relationship between events and media, enable event discovery, and perform event-based media tagging, indexing, and retrieval. Despite the outstanding work done in this area, a major problem that remains open is how to infer the link between visual concepts and events. In particular, the possibility of understanding which perceptual elements allow a human recognizing the event depicted by an image would open new directions in event media discovery. In this paper we introduce the concept of event saliency to define the above event-revealing perceptual elements, and we propose an original method to detect it by exploiting crowd knowledge through gamification. We propose an adversarial game with a hidden purpose, where users are engaged in competitive roles: masking photos to prevent competitors recognizing the related event, and discovering events in photos masked by other players. Rules and incentives are defined to minimize cheating and force players to focus on details that really matter. A suitable algorithm composes the masks created by different players on the same media, thus producing a saliency map that, different from the traditional concept of saliency, does not focus on perceptual prominence but rather on event-related semantics of media. A thorough validation on public datasets is presented, and initial experiments to apply event saliency in detection tasks are proposed . Furthermore, an event-saliency dataset is disclosed to allow further research.