Title :
A mathematical model for crawler revisit frequency
Author :
Dixit, Ashutosh ; Sharma, A.K.
Author_Institution :
Dept. of Comput. Enginnering, YMCA Inst. of Eng., Faridabad, India
Abstract :
WWW´s expansion coupled with high change frequency of web pages poses a challenge for maintaining and fetching up-to-date information. The traditional crawling methods are no longer catch up with this updating and growing web. Alternative distributed crawling scheme that uses migrating crawlers try to maximize the network utilization by minimizing the network load but are hampered due to the deficiency in their web page refresh techniques. The absence of effective measures to verify whether a web page has been changed or not is another challenge. In this paper, an efficient approach for computing revisit frequency is being proposed. Web pages which frequently undergo up-dation are detected and accordingly revisit frequency for the pages is dynamically computed.
Keywords :
Web sites; search engines; World Wide Web; alternative distributed crawling scheme; crawler revisit frequency; crawling methods; information fetching; network utilization; web page refresh techniques; Crawlers; Frequency; Maintenance engineering; Mathematical model; Search engines; Uniform resource locators; Web pages; Web search; Web server; World Wide Web; Frequency of revisit; Mobile Crawler; Search engine; World Wide Web;
Conference_Titel :
Advance Computing Conference (IACC), 2010 IEEE 2nd International
Conference_Location :
Patiala
Print_ISBN :
978-1-4244-4790-9
Electronic_ISBN :
978-1-4244-4791-6
DOI :
10.1109/IADCC.2010.5422936