The Influence of Relevance Levels on the Effectiveness of Interactive Information Retrieval

In this paper, we focus on the effect of graded relevance on the results of interactive information retrieval (IR) experiments based on assigned search tasks in a test collection. A group of 26 subjects searched for four Text REtrieval Conference (TREC) topics using automatic and interactive query expansion based on relevance feedback. The TREC- and user-suggested pools of relevant documents were reassessed on a four-level relevance scale. The results show that the users could identify nearly all highly relevant documents and about half of the marginal ones. Users also selected a fair number of irrelevant documents for query expansion. The findings suggest that the effectiveness of query expansion is closely related to the searchers’ success in retrieving and identifying highly relevant documents for feedback. The implications of the results on interpreting the findings of past experiments with liberal relevance thresholds are also discussed.