Reducing web crawler overhead using mobile crawler

To search any information on the web users extensively use the search engines. As the growth of the World Wide Web exceeded all expectations, the search engines rely on web crawlers to maintain the index of billions of pages for efficient searching. The web crawlers have to interact with millions of hosts and retrieve the pages continuously to keep the index up-to-date. It is reported in the literature that the 40% of the current Internet traffic and bandwidth consumption is due to these crawlers. These crawlers also cause load on the remote server by using its CPU cycles and memory. In addition, I/O performance, network resources, and OS limits must be taken into account in order to achieve high performance at a reasonable cost. This paper address all those problems by proposing a system based on mobile crawlers using mobile agent. The proposed approach employs mobile agents to crawl the pages. These mobile crawlers identify the modified pages at the remote site without downloading them instead it downloads those pages only, which have actually been modified since last crawl. Hence it will reduce the Internet traffic and load on the remote site considerably. This system can be implemented by the help of java aglets.