Mining Visual Knowledge for Multi-Lingual Image Retrieval

Users commonly rely just on scarce textual annotation when their searches for images are semantic or conceptual based. Rich visual information is often thrown away in basic annotation-based image retrieval because its relationship to the semantic content is not always clear. To ensure that appropriate visual information is included, we propose using visual clustering within pre-processing and post-processing steps of text-based retrieval. A clustering algorithm finds pairs of images that are nearly identical and are, therefore, presumed semantically similar. The output from basic retrieval systems is a ranked list of images based only on lexical term matching. The obtained cluster knowledge is then used to modify the ranking result during the post-processing step. Low ranked images considered nearly identical to more highly ranked images are then pulled up. The modularity of this architecture allows us to integrate a data mining process without having to change core information retrieval systems. Evaluation on a cross-language image retrieval test collection showed that this method improved retrieval performance for certain queries in multilingual settings.