Probabilistic retrieval with a visual grammar

We describe a system for content-based retrieval and classification of multispectral images. Our system models images on pixel, region and scene levels. To reduce the gap between low-level features and highlevel user semantics, and to support complex query scenarios that consist of many regions with different feature characteristics, we propose a probabilistic visual grammar that includes automatic identification of region prototypes and modeling of their spatial relationships. A Bayesian framework is used to automatically classify scenes based on these models. We demonstrate our system with query scenarios that cannot be expressed by traditional region or scene level approaches but where the visual grammar provides accurate classifications and effective retrieval.