ComRank: metasearch and automatic ranking of XML retrieval system

Different information retrieval (IR) systems often return very diverse results lists for the same query. This is problematic for users since no one IR system works best for every scenario, and it is difficult for the user to know which system will work best a priori. The challenge of metasearch is to merge results lists from several IR systems, with the goal of outperforming each of the constituent systems. This paper presents ComRank, a metasearch system that discriminates in favour of results that (1) originate by consensus amongst several systems; (2) are highly ranked in their original systems and (3) originate from the better performing systems. Importantly, ComRank determines the `better´ performing systems without the need for human judgements. Rather, it uses an automatic assessment process that ranks systems by their pseudo-relevance, as derived from highly ranked results in ComRank´s list. We apply our methods to the INEX Collection, which is an unexplored domain for these methods, and show that they are comparable to or better than baseline alternatives