Center Block Duplication Detection and Indexing for Efficient Web Information Retrieval

Duplicated information in today´s Web has serious negative impact to search engines in that it increases the size of the index and results in low efficiency for Web information retrieval. A large amount of Web content duplication happens at block level in addition to page level. Besides, when searching the web, in most cases the desired information is located at the center block of a page. Based on these two observations, we propose a block level duplication detection algorithm, and index center blocks instead of entire Web pages for Web information retrieval. Experiments show that these strategies can effectively reduce index size and indexing time without sacrificing the effectiveness of Web information retrieval.