A service crawler framework for similarity based web service discovery

Due to the explosive growth in availability of Web services over the open Web and in heterogeneous sources in which they are available, discovering relevant web services for a given task continues to be challenging. In order to deal with these problems, a bottom-up approach based on finding published service descriptions for developing a web service discovery framework was proposed. We discuss in detail the service crawler employed to search and find published WSDL service descriptions on the Web. The service descriptions are semantically analyzed in order to improve efficiency of the discovery process as per user context during keyword search. Since the service crawler will periodically make repeated runs to find new service descriptions or to check the status of already crawled services, the framework is of an inherently dynamic nature. Hence, it is critical to keep track of various entities like visited URLs, already added services and successfully processed service descriptions to avoid rework when the service data set changes, for which an incremental change propagation strategy was incorporated.