Combining content and context information for semantic image analysis and classification

In this paper, a learning approach to semantic image analysis and classification is proposed that combines global and local information, with explicitly defined knowledge in the form of an ontology. The ontology specifies the selected domain, its sub-domains, the concepts related to each sub-domain as well as contextual information. Support Vector Machines (SVMs) are employed in order to provide image classification to one of the defined sub-domains based on global image descriptions and, after a segmentation algorithm is applied, to perform an initial mapping between region low-level visual features and the concepts in the ontology. Then, a decision function, that receives as input the region to concepts associations together with contextual information, realizes image classification based on local-level information. The contextual information used is in the form of frequency of appearance of each concept in every particular sub-domain. A fusion mechanism combines the intermediate classification results, provided by the local-and global-level information processing, and decides on the final classification. A Genetic Algorithm (GA) is employed for optimizing the fusion process. Experiments with images from the personal collection domain demonstrate the performance of the proposed approach.