Architecture Design of Subject-Oriented Web Crawler

In response to the defects of traditional search engines of which it will return a large amount of information when users search keywords, and it´s hard to get the useful information, thus propose the thought of subject division, that is the subject-oriented search engine.Subject crawler is the key and unique components of the subject-based search engine. The structure of the crawler has a significant impact on the speed of Web resources, as well as the multi-machine distributed extended functionality. This paper studies and designed an architecture of subject crawler, which has flexible modular scalability and multi-machine distributed scalability, and elaborated it.