Joint Optimization of Segmentation and Color Clustering

Binary energy optimization is a popular approach for segmenting a color image into foreground/background regions. To model the appearance of the regions, color, a relatively high dimensional feature, should be handled effectively. A full color histogram is usually too sparse to be reliable. One approach is to explicitly reduce dimensionality by clustering or quantizing the color space. Another popular approach is to fit GMMs for soft implicit clustering of the color space. These approaches work well when the foreground/background are sufficiently distinct. In cases of more subtle difference in appearance, both approaches may reduce or even eliminate foreground/background distinction. This happens because either color clustering is performed completely independently from the segmentation process, as a preprocessing step (in clustering), or independently for the foreground and independently for the background (in GMM). We propose to make clustering an integral part of segmentation, by including a new clustering term in the energy function. Our energy function with a clustering term favours clusterings that make foreground/background appearance more distinct. Thus our energy function jointly optimizes over color clustering, foreground/background models, and segmentation. Exact optimization is not feasible, therefore we develop an approximate algorithm. We show the advantage of including the color clustering term into the energy function on camouflage images, as well as standard segmentation datasets.