Activity Discovery from Surveillance Videos

Multi-agent interactions often result in mutual occlusion sequences which constitute a visual signature for the event. We define six qualitative occlusion primitives based on the persistence hypothesis (objects continue to exist even when hidden from view): isolated, occlude with foreground, occlude by background, disappear, enter and exit. Variable length temporal sequences of occlusion primitives are shown to be useful features for categorizing many classes of semantically significant events. Occlusion primitive labels depend on agent positions in the image, which are determined by combining foreground blob tracking and image motion. No prior knowledge of domain or camera calibration is necessary. New foreground blobs are identified as putative agents which may undergo occlusions, split into multiple agents, merge back again, etc. Transition sequences are mined to identify semantic categories (e.g., people disembarking from a vehicle involve a series of splits). Occlusion features alone may be useful for distinguishing some broad categories of interaction states, and together with features such as agent shape and motion histories, these form a rich signature for different event types that can be classified without camera calibration or any environment/agent/action model priors