An Improved Algorithm of STC for the Deletion of Duplicated Web pages Based on Repeated Strings

This paper proposes an improved algorithm of STC for deleting duplicated Web pages based on repeated strings. The main method of the algorithm is to extract repeated character strings. The repeated strings are used as the mark of each phrase in order to build the suffix tree. This is mapped onto the inverse index in order to enable the STC algorithm to delete duplication. The algorithm also aims to reduce the errors made by the existing algorithms for deletion. Experimental results indicate that the improved algorithm has a better rate of accuracy and good temporal and spatial characteristics.