Taxonomy of Directing Semantics for Film Shot Classification

The immense indexing potential of motion cues has hitherto been realized only in domains with more apparent structure (e.g., sport videos). To address the lack of theoretical attention and to realize the potential of motion-based indexing in the subtler film domain, we propose a systematic approach to build taxonomy for film directing semantics. These motion-related semantics are grounded upon cinematography and are thus more appealing to users. In order to automate the classification of these semantics, we have developed a novel Markov random field based motion segmentation algorithm with an integral foreground/background identification capability based on edge occlusion reasoning. This algorithm is sufficiently robust and fast for film domain conditions, and allows us to formulate salient and novel motion descriptors capable of mapping to the proposed directing semantics. We demonstrate the validity of the framework and effectiveness of the motion-based descriptors by classifying shots from Hollywood domain movies according to the proposed taxonomy with satisfactory results.