Query log simulation for long-term learning in image retrieval

In this paper we formalise a query simulation framework for the evaluation of long-term learning systems for image retrieval. Long-term learning relies on historical queries and associated relevance judgements, usually stored in query logs, in order to improve search results presented to users of the retrieval system. Evaluation of long-term learning methods requires access to query logs, preferably in large quantity. However, real-world query logs are notoriously difficult to acquire due to legitimate efforts of safeguarding user privacy. Query log simulation provides a useful means of evaluating long-term learning approaches without the need for real-world data. We introduce a query log simulator that is based on a user model of long-term learning that explains the observed relevance judgements contained in query logs. We validate simulated queries against a real-world query log of an image retrieval system and demonstrate that for evaluation purposes, the simulator is accurate on a global level.