Title :
Resolving domains in large scale Web crawling
Author_Institution :
Sch. of Software Eng., Huazhong Univ. of Sci. & Technol., Wuhan, China
Abstract :
Efficient domain resolving is essential for large scale Web crawl. Based on batch processing, data structure and algorithms are presented for maintaining domains and addresses in crawling, and their performances are analyzed mathematically. Large scale domain resolving system is designed with proposed data structure. The theoretical analysis and experiments show that the speed of several thousand links per second for billions of links or hundreds of millions hosts can be achieved on one common personal computer.
Keywords :
Internet; search engines; arge scale Web crawl; batch processing; data structure; large scale domain resolving system; personal computer; Algorithm design and analysis; Crawlers; Data structures; Maintenance engineering; Merging; Random access memory; Web sites; batch-based information maintenance; domain resolving; web crawling;
Conference_Titel :
Computer Science and Service System (CSSS), 2011 International Conference on
Conference_Location :
Nanjing
Print_ISBN :
978-1-4244-9762-1
DOI :
10.1109/CSSS.2011.5974572
