Combining Multiple Retrieval Systems Using Combinatorial Fusion Analysis and Rank-Score Characteristic Function

Combining the resulting lists of multiple information retrieval (IR) systems has been known to outperform, in many cases, the best of the individual systems. However, it remains a challenging question to know what combination method to use and in what conditions the combination system can perform better than its individual systems. In this paper, we use an information fusion paradigm: Combinatorial Fusion Analysis (CFA) to study these issues. We take the TREC dataset as our experiment data and use the rank-score characteristic (RSC) function to measure the cognitive diversity between different individual systems. Results from our experiment demonstrate that: 1) combined system can improve performance only if the individual systems have relative good performance and are diverse, 2) there is no guarantee that the combined system performs better when more individual systems are added, and 3) rank combination is better than score combination in majority of the cases when the diversity between two individual systems measured by the RSC function is large enough.