Relevance Judgments for Web Services Retrieval - A Methodology and Test Collection for SWS Discovery Evaluation

Semantic web services (SWS) promise to take service oriented computing to a new level by allowing to automatically locate and use functionality exposed as web services. At the core of SWS are solutions to the problem of SWS discovery, i.e., the problem of comparing semantic goal descriptions with semantic offer descriptions to determine services relevant to a given request. A plethora of different approaches to this problem have been proposed, but their comparative evaluation is challenging.While the evaluation setups from the information retrieval (IR) community provided a natural starting point to SWS discovery evaluation, the applicability of their assumptions have not been sufficiently discussed so far. This paper discusses the differences between traditional IR and SWS discovery and the resulting fundamental problems. It proposes a methodology and experimental setup for SWS discovery evaluation that addresses these problems. An initial realistic service test collection is presented and issues related to the central notion of relevance in the context of service discovery are examined. In particular the consistency of relevance judgments using three different relevance scales is experimentally investigated and the consequences on the evaluation methodology are discussed.