Research on a dynamic adjust crawling algorithm for guiding the topic crawler through Tunnels

The problem of Tunnels is always the focus of topic crawler. Based on the study of VSM, the paper added the impact of the text structure of web documents to the topic similarity, improved VSM text classification algorithm to make the prediction more accurate, and applied to the dynamic adjustment topic crawler algorithm through the tunnel. By analyzing the influence by features of Web Community and tunneling, taking the genetic factors of parent page and child pages into account, applied to the web page similarity calculation. In order to improve the shortcomings of the traditional tunnel method, this paper designed a new algorithm to make crawler dynamically adjust the K values according to the corresponding calculated strategy during crawling the pages, Making Web Community and tunnels to form a relatively complete thematic clusters to improve the web crawl rate.