Semantic segmentation as image representation for scene recognition

We introduce a novel approach towards scene recognition using semantic segmentation maps as image representation. Given a set of images and a list of possible categories for each image, our goal is to assign a category from that list to each image. Our approach is based on representing an image by its semantic segmentation map, which is a mapping from each pixel to a predefined set of labels. Among similar high-level approaches, ours has the capability of not only representing what semantic labels the scene contains, but also their shapes, sizes and locations. We also investigate the effect of varying experiment parameters, including varying labels used, semantic segmentation technique, and semantic training source. We obtain state-of-the-art results over Siftflow and MSRC-21 datasets.