Object-based scene segmentation combining motion and image cues

This paper presents an object-based scene segmentation algorithm which combines the temporal information (e.g. motion) from video and image cues from individual frame. First a motion-based segmentation is decided based on the hierarchical principal component split (HPCS) algorithm for multi-moving-object motion classification. HPCS is a binary-tree-structured recursive procedure which clusters the feature blocks according to their principal component of the feature track matrix. Tracking of feature blocks from multiple frames (⩾2) can be effectively processed and this results in a more accurate rigid motion classification. Experimental result shows that by using motion alone, some mostly homogeneous blocks may fit well to more than one motion classes so that ambiguity occurs. Such blocks are categorized into the so-called “undetermined” region (or U-region) for further processing. An image segmentation scheme using local pixel statistics of blocks in the U-region (U-blocks) is applied to find “valid voting regions” (VVRs). A VVR has a mostly homogeneous interior and is surrounded by a closed contour consisting of relatively high gradient points, which can offer the needed discriminating power for classifying each VVR to its belonging object class by motion voting. By combining the motion-based segmentation with the classification result of VVRs, the final object-based scene segmentation is determined. Simulation results are presented