Multi-Dimensional Evaluation of Information Retrieval Results

Evaluation of information retrieval (IR) results is an important and difficult activity. Available evaluation methods are commonly based on the classical recall and precision measures. While these methods provide an evaluation of average system performance, they are not able to identify influence of other factors such as retrieval task and system user. Detailed information is hidden behind the average and no indication can be obtained from the evaluation for further improvement. Therefore, there is a need of new methods for detailed evaluation. In this paper, we propose a multi-dimensional approach to the evaluation of IR results so that effects of both an IR system and the environment where the system performs on retrieval results can be examined. It aids in the identification of problems of the system and the environment with respect to retrieval effectiveness, and assists in making improvement for IR.