An efficient scheme to remove crawler traffic from the Internet

Author

Yuan, X. ; MacGregor, M.H. ; Harms, J.

Author_Institution

Dept. of Comput. Sci., Alberta Univ., Edmonton, Alta., Canada

fYear

2002

fDate

14-16 Oct. 2002

Firstpage

90

Lastpage

95

Abstract

We estimate that approximately 40% of current Internet traffic is due to Web crawlers retrieving pages for indexing. We address this problem by introducing an efficient indexing system based on active networks. Our approach employs strategically placed active routers that constantly monitor passing Internet traffic, analyze it, and then transmit the index data to a dedicated back-end repository. Our simulations have shown that active indexing is up to 30% more efficient than the current crawler-based techniques.

Keywords

Internet; database indexing; telecommunication network routing; telecommunication traffic; Internet traffic; Web crawlers; active indexing; active networks; active routers; crawler traffic removal; dedicated back-end repository; efficient indexing system; index data transmission; simulations; traffic analysis; traffic monitoring; Computer networks; Crawlers; Indexing; Internet; Monitoring; Proposals; Search engines; Switches; Telecommunication traffic; Web pages;

fLanguage

English

Publisher

ieee

Conference_Titel

Computer Communications and Networks, 2002. Proceedings. Eleventh International Conference on

ISSN

1095-2055

Print_ISBN

0-7803-7553-X

Type

conf

DOI

10.1109/ICCCN.2002.1043051

Filename

1043051