Traffic Adaptive Optimum Updating Scheme for Search Engines

The increasing complexity, heterogeneity and dynamism of Web and its applications have made Web information retrieval less recent, less relevant and unmanageable. The search engines face the problem of keeping the contents of its repository consistent with the pages present in the global database using optimum resource utilization. This paper proposes a traffic adaptive optimum updating scheme (TAOS) to eliminate the needless requests of Web crawlers in updating the search engine repository. The scheme also incorporates partial upload of the updated document to the search engine repository. A self-managing autonomic computing architecture is proposed to regulate the load on network bandwidth and Web servers. The proposed updating scheme is compared for the freshness of search engine repository with the page refresh policies used by Web crawlers. The load on network bandwidth and Web servers are also analysed for effective resource utilization and is compared with the one consumed during crawler updating.