DocumentCode :
2732612
Title :
Reusability assessment of test collections with multi-levels of judgments
Author :
Khodabakhsh, Maryam ; Araban, Saeed
Author_Institution :
Dept. of Comput., Ferdowsi Univ. of Mashhad, Mashhad, Iran
fYear :
2012
fDate :
18-19 Oct. 2012
Firstpage :
215
Lastpage :
220
Abstract :
Constructing good test collection is an expensive and time-consuming process. Traditionally, test collections contain binary judgments. In recent years, however, there has been increasingly interest in test collections with Multi-levels judgments and of certain qualities. Such collections are even more expensive to construct. Therefore, ability to reuse test collections can not only save construction costs, but also boosts our confidence in their quality. This paper proposes a method for assessing reusability of a test collection with multi-level judgments. The proposed method can help IR researchers to determine whether an existing test collection with a set of multi-level judgments is suitable for evaluating a new IR system or not. Results of our experiments (on MAHAK test collection) suggest that this method can help assessing reusability of a test collection.
Keywords :
data handling; information retrieval systems; IR system; MAHAK test collection; binary judgment level; information retrieval; multilevel judgment; reusability assessment; test collection; Computers; Logistics; Measurement; Support vector machines; Testing; Uncertainty; Vectors; confidence interval; evaluation; information retrieval (IR) system; multi-level judgments; reusability; test collection;
fLanguage :
English
Publisher :
ieee
Conference_Titel :
Computer and Knowledge Engineering (ICCKE), 2012 2nd International eConference on
Conference_Location :
Mashhad
Print_ISBN :
978-1-4673-4475-3
Type :
conf
DOI :
10.1109/ICCKE.2012.6395381
Filename :
6395381
Link To Document :
بازگشت