Scalability and reliability in a distributed search engine

Author

Sato, Nobuyoshi ; Udagawa, Minoru ; Uehara, Minoru ; Sakai, Yoshifumi ; Mori, Hideki

Author_Institution

Dept. of Inf. & Comput. Sci., Toyo Univ., Saitama, Japan

fYear

2002

fDate

17-20 Dec. 2002

Firstpage

57

Lastpage

62

Abstract

We have developed a distributed search engine, called cooperative search engine (CSE), in order to retrieve fresh information. In CSE, a local search engine located in each Web server makes an index of local pages. A meta search server integrates these local search engines in order to realize a global search engine. In such a way, the communication delay occurs at retrieval time. So, it is thought to be difficult to search quickly. However we have developed several speedup techniques in order to realize real time retrieval. In addition, the meta server is a single point of failure in CSE. So, we propose redundancy of meta search servers in order to increase availability of CSE. In this paper we describe scalability and reliability of CSE and their evaluations.

Keywords

Internet; file servers; information retrieval; reliability; search engines; Web server; communication delay; cooperative search engine; distributed search engine; local page indexing; local search engine; meta search server; real time retrieval; redundancy; reliability; scalability; speedup techniques; Delay effects; Distributed computing; Information retrieval; Metasearch; Network servers; Redundancy; Scalability; Search engines; Web pages; Web server;

fLanguage

English

Publisher

ieee

Conference_Titel

Parallel and Distributed Systems, 2002. Proceedings. Ninth International Conference on

ISSN

1521-9097

Print_ISBN

0-7695-1760-9

Type

conf

DOI

10.1109/ICPADS.2002.1183378

Filename

1183378