A Hierarchical Trajectory-Based Representation for Video

This paper proposes the Trajectory Tree as a hierarchical representation for video sequences containing multiple layers of resolution and various degrees of spatio-temporal consistency. We present a technique for generating an initial set of trajectories through partition tracking. Trajectories are formed of temporally linked regions homogeneous in color and coherent in terms of affine motion models. Next, merging algorithms exploiting homogeneity criteria over multiple frames are applied towards the construction of hierarchical representations. Mechanisms for guaranteeing spatio-temporal continuity are introduced, including temporal splitting and long-term color-based merging. The final hierarchy is constructed with the sequential application of novel long-term affine-based and translation-based motion similarity measures. The use of global motion characteristics allows the consistent segmentation and tracking of moving objects and their sub-regions, including cases of non-rigid motion and objects which stop moving for several frames. Experimental results are presented for the Table Tennis, Foreman, Coastguard, Mother and Daughter and Speedway sequences.