Change propagation based incremental data handling in a Web service discovery framework

Due to the explosive growth in availability of Web services over the open Web and the heterogeneous sources in which they are available, discovering relevant web services for a given task continues to be challenging. In order to deal with these problems, a bottom-up approach based on finding published service descriptions to automatically build a service repository for developing a web service discovery framework was proposed. This framework employs a Service Crawler to find published service descriptions on the Web. Since the service crawler will periodically make repeated runs to find new service descriptions or to check the continued availability of already crawled services, the framework is of an inherently dynamic nature. Hence, it is critical to keep track of various entities like visited URLs, already added & successfully processed service descriptions to avoid rework when the service data set changes. In order to cope with this problem, we developed a change propagation technique based on an event based state machine to incorporate an incremental processing strategy in this Web scale framework.