DocumentCode :
1627361
Title :
A mathematical model for crawler revisit frequency
Author :
Dixit, Ashutosh ; Sharma, A.K.
Author_Institution :
Dept. of Comput. Enginnering, YMCA Inst. of Eng., Faridabad, India
fYear :
2010
Firstpage :
316
Lastpage :
319
Abstract :
WWW´s expansion coupled with high change frequency of web pages poses a challenge for maintaining and fetching up-to-date information. The traditional crawling methods are no longer catch up with this updating and growing web. Alternative distributed crawling scheme that uses migrating crawlers try to maximize the network utilization by minimizing the network load but are hampered due to the deficiency in their web page refresh techniques. The absence of effective measures to verify whether a web page has been changed or not is another challenge. In this paper, an efficient approach for computing revisit frequency is being proposed. Web pages which frequently undergo up-dation are detected and accordingly revisit frequency for the pages is dynamically computed.
Keywords :
Web sites; search engines; World Wide Web; alternative distributed crawling scheme; crawler revisit frequency; crawling methods; information fetching; network utilization; web page refresh techniques; Crawlers; Frequency; Maintenance engineering; Mathematical model; Search engines; Uniform resource locators; Web pages; Web search; Web server; World Wide Web; Frequency of revisit; Mobile Crawler; Search engine; World Wide Web;
fLanguage :
English
Publisher :
ieee
Conference_Titel :
Advance Computing Conference (IACC), 2010 IEEE 2nd International
Conference_Location :
Patiala
Print_ISBN :
978-1-4244-4790-9
Electronic_ISBN :
978-1-4244-4791-6
Type :
conf
DOI :
10.1109/IADCC.2010.5422936
Filename :
5422936
Link To Document :
بازگشت