The Influence of Document Presentation Order and Number of Documents Judged on Users’ Judgments of Relevance

This article attempts to verify the hypothesis of the document presentation order by an empirical, two-stage experiment. It aims to identify the relationship between number of documents judged and order effect. The results indicate that significant order effect takes place when 15 and 30 documents are presented. Sets with 45 and 60 documents still reveal the order effect. However, subjects are not influenced by order of presentation when the set of documents has 5 and 75 members, respectively