A Multi-Agent Simulation Framework for Spiders Traversing the Semantic Web

Although search engines traditionally use spiders for traversing and indexing the Web, there has not yet been any methodological attempt to model, deploy and test learning spiders. The flourishing of the semantic Web provides understandable information that may improve the accuracy of search engines. In this paper, we introduce BioSpider, an agent-based simulation framework for developing and testing autonomous, intelligent, semantically-focused Web spiders. BioSpider assumes a direct analogy of the problem at hand with a multi-variate ecosystem, where each member is self-maintaining. The population of the ecosystem comprises cooperative spiders incorporating communication, mobility and learning skills, striving to improve efficiency. Genetic algorithms and classifier rules have been employed for spider adaptation and learning. A set of experiments has been performed in order to qualitatively test the efficacy and applicability of the proposed approach