Semantic classification by covariance descriptors within a randomized forest

This paper investigates an approach to perform semantic classification in aerial imagery by compactly integrating multiple feature cues, like appearance and 3D height information. We therefore propose a novel technique to incorporate powerful covariance region descriptors into the decision nodes of a randomized forest framework efficiently. The concept of finding reliable binary splits is based on repeated random sampling of distributions that are specified by mean vectors and covariance matrices. The sampling strategy is related to Monte Carlo simulations and perfectly fits the learning strategy of randomized decision trees, while the covariance descriptors are exploited to perform a plausible feature cue integration. To show state-of-the-art performance, we first evaluate our proposed approach on the MSRC dataset including 21 object classes. Then, we illustrate how an additional integration of 3D information improves the classification accuracy in real world aerial images taken from Dallas, San Francisco, and Graz. In addition, we use the available camera data and 3D information to combine the overlapping per-image classifications into a large-scale semantic description map that is directly applicable to virtual or procedural 3D modeling of urban environments.