A dynamic reconfiguration model for a distributed web crawling system

Author

Hongfei, Yan ; Jianyong, Wang ; Xiaoming, Li

Author_Institution

Dept. of Comput. Sci. & Technol., Peking Univ., Beijing, China

fYear

2001

fDate

2001

Firstpage

157

Lastpage

162

Abstract

A web crawling system employing a parallel and distributed architecture needs to have a mechanism to bring the whole system in a coordinated state when the nodes are added to or removed from the system. This paper presents an efficient dynamic reconfiguration model that can be used in such a system. The study shows that this model leads to some nice properties, such as load balance and low traffic in the system, which contribute to high performance. Currently this model is being implemented in WebGather, a well-known Chinese and English web search engine

Keywords

Internet; information resources; parallel architectures; search engines; Chinese and English web search engine; WebGather; distributed architecture; distributed web crawling system; dynamic reconfiguration model; load balance; parallel architecture; Computer architecture; Computer networks; Computer science; Crawlers; Laboratories; Search engines; Service oriented architecture; Telecommunication traffic; Traffic control; Web search;

fLanguage

English

Publisher

ieee

Conference_Titel

Computer Networks and Mobile Computing, 2001. Proceedings. 2001 International Conference on

Conference_Location

Los Alamitos, CA

Print_ISBN

0-7695-1381-6

Type

conf

DOI

10.1109/ICCNMC.2001.962591

Filename

962591