A Novel Language-Model-Based Approach for Image Object Mining and Re-ranking

One leading framework for image object mining is the bag-of-words (BOW) approach. The idea is to encode an image as a collection of visual words of the quantized local patches. Objects in the image can then be retrieved through inferring the semantic topics associated with the set of visual words. However, the visual BOW mining framework is apt to suffer from the so-called term-mismatch problem (a.k.a. vocabulary problem). This is caused by the poverty of query information, and consequently becomes an obstacle to deal with synonymy (i.e., different visual words for describing the same object). In this paper, we propose a novel language-model-based approach with pseudo-relevance feedback for addressing the vocabulary problem in visual BOW mining. We employ the pseudo positive images produced in response to the original query as a set of "cues" to gradually refine the query language model. Unlike traditional approaches that only ruggedly append feedback information into the original query, the proposed approach reconstructs the query language model with finer granularities so that the query concepts can be captured more accurately. The proposed approach is experimentally evaluated using two different types of image object databases. Our algorithms are shown to bring significant improvement in the retrieval accuracy over a non-feedback baseline, and achieve better performance than conventional feedback approaches.