An Architectural Framework of a Crawler for Locating Deep Web Repositories Using Learning Multi-agent Systems

The World Wide Web (WWW) has become one of the largest and most readily accessible repositories of human knowledge. The traditional search engines index only surface Web whose pages are easily found. The focus has now been moved to invisible Web or hidden Web, which consists of large warehouse of useful data such as images, sounds, presentations and many other types of media. To utilize such data, there is a need for specialized program to locate those sites as we do with search engines. This paper discusses about an effective design of a hidden Web crawler that can autonomously discover pages from the hidden Web by employing multi-agent Web mining system. A theoretical framework is suggested to investigate the resource discovery problem and the empirical results suggest substantial improvement in the crawling strategy and harvest rate.