What to Reuse?: A Probabilistic Model to Transfer User Annotations in a Surveillance Video

Techniques to extract or understand interactions between moving objects in video is becoming increasingly important as the amount of video increases. Applications in surveillance range from understanding traffic to studying fish schooling behaviour. Because of the massive amount of data, fast, approximate techniques based on statistical models are common. These models connect user annotations (labels) to scenes in a (short) video segment. The connection forms a domain, which associates information about moving objects in scenes with the labels, such as to indicate whether a user considers a particular traffic scene to be "dangerous." Unfortunately a statistical model trained in one domain often yields low precision and recall when applied to another domain because the random variables that explain video content exhibit changing marginal and conditional probability distributions over time (e.g., due to different backgrounds, changes in illumination, shading, and numbers of moving objects). This problem is exacerbated when new domains continuously arise (e.g., in the real-time processing of video) and user annotations are only limited to training data, a common scenario for surveillance video. In this paper, we propose a new, cross-domain technique that reuses labelled content from source domains to improve the prediction of user annotations in a target domain. Our model probabilistically learns how users annotate scenes based on the similarity of target to source domains. Two domains that are similar will share a large number of observable features. We encode the similarity in a covariance matrix, which flexibly allows allows users to set an arbitrary covariance structure between pairs of domains before training the model. Experiments show that our method improves state-of-the-art techniques (SVM and CF) in predicting dangerous scenes in real-world traffic surveillance videos.