Enhancing focused crawling with genetic algorithms

Web crawlers are one of the most crucial components in search engines and their optimization would have a great effect on improving the searching efficiency. In this paper, we introduce an intelligent crawler called Gcrawler that uses a genetic algorithm for improving its crawling performance. Gcrawler estimates the best path for crawling on one hand and expands its initial keywords by using a genetic algorithm during the crawling on the other hand. This is the first crawler that acts intelligently without any relevance feedback or training. All the processes are online and there is no need for direct interaction with the users.