A Perceptual Depth Shape-based CRF Model for Deformable Surface Labeling

Real-time deformable scene understanding is a challenging task. In this paper, we address this problem by using Conditional random fields (CRFs) framework and perceptual shape salience occupancy patterns. CRF is a powerful probabilistic model that has been widely used for labelling image segments. It is particularly well-suited to modelling local interactions and global consistency among bottom-up regions (e.g. super pixels). However, its capacity could be limited if the underlying feature potentials are not well reflecting the scene properties. We propose a depth shape-based CRF model for deformable surface (sand in our case) labelling by utilizing expressive novel shape salience occupancy patterns (SOP). Experimental results demonstrate the effectiveness and robustness of the method on recorded video datasets. While our work has concentrated on sand surface labelling, the approach can be applied to other surface materials (e.g. snow, mud), and extended to non-planar surfaces as well (e.g. sculpting blocks).