Title :
Research on Tunnel Phenomenon in Focused Crawler
Author :
Yan Chun ; Hang, Zhang
Author_Institution :
Comput. Sci. & Technol. Sch., Wuhan Univ. of Technol., Wuhan, China
Abstract :
With the rapid development of search engine technology, the technology of focused crawler has been widely concerned and deeply researched. In this research field, there is a "tunnel phenomenon" seriously affecting the crawler\´s efficiency and coverage. This paper proposes a simple algorithm on correlation calculation and a good method on dynamic adjustment of the threshold, which can effectively solve the problems brought by the tunnel phenomenon. The method works well verified by the experiments.
Keywords :
Internet; search engines; software agents; dynamic threshold adjustment; focused crawler; search engine technology; tunnel phenomenon; Algorithm design and analysis; Conferences; Correlation; Crawlers; Heuristic algorithms; Search engines; Strontium;
Conference_Titel :
Internet Technology and Applications, 2010 International Conference on
Conference_Location :
Wuhan
Print_ISBN :
978-1-4244-5142-5
Electronic_ISBN :
978-1-4244-5143-2
DOI :
10.1109/ITAPP.2010.5566176
