Algorithm for detecting dynamic webpage and its importance

During web search using crawling, indexing, relevance it is found that there exist many duplicate web-pages with different URLs, these URLs are normalized when used by crawler. Many web-pages are found to be dynamic, for which different web contents are found with the same URL, during different instances of searches. In this paper, we discuss about the necessity to detect these dynamic web-pages and propose an algorithm to identify this dynamism. The normalization of URLs can be done using various methods explained in [1], [2] & [7], or using the DUST algorithm [3] but it is necessary first to identify the dynamic web-page before normalization. After implementing the proposed algorithm with DUST rule it is expected that the detection rate of dynamic web-pages improves, resulting in reduction of the time spent for crawling, indexing etc.