Sub-Shots - Basic Units of Video

Shots are commonly regarded as the basic unit of video above the frame level. Scenes and higher temporal structures are usually based upon them. However, the common definitions of shots and scenes do not completely support this hierarchy. Scene boundaries can occur within shots. Reasons and examples are given in this paper. A new level in the hierarchy of temporal video segments named sub-shots is proposed. Sub-shots are parts of shots. They are restricted to one event and limited to only a small change in visual content. The determination of scenes can then be based upon sub-shots. Sub-shots can also be used for key-frame extraction. Two algorithms, a clustering approach based on visual content and a motion analysis algorithm are proposed and yield experimentally promising results.