Evaluation of Open-Domain Question Answering Systems within a Time Constraint

Previous works on evaluating the performance of Question Answering (QA) systems are focused in the evaluation of the precision. Nevertheless, the importance of the answer time never has been evaluated. This paper studies the problem of the evaluation in QA systems, focusing on the way in which the time can be considered. Specifically, we carried out an exercise for the evaluation of QA systems within a time constraint in the CLEF-2006 campaign, proposing new measures which combine the precision with the answer time. Also, the performance achieved by each participant system and statistical analysis of the results is given. The main conclusion is that the evaluation of QA systems in realtime can be a more realistic scenario than the traditional used for the main evaluations forums in QA.