A research on a defending policy against the Webcrawler´s attack

With the increasing of the amount of Internet information, there are different kinds of Web crawlers fetching information from websites at anytime, anywhere, some of which are fetching information normally and some are attacking Web sites based upon application level and then causing the breakdown of servers, For a Web site, how to distinguish different kinds of crawlers effectively and accurately prevent bad crawlers´ attack, or decrease crawlers´ burden on the Web site during the rush hour becomes an important case, This paper based on a deep analysis of Web crawler´ s fetching process, summarized the Web crawlers´ threats on the security of Web sites and then provide a trap policy and defend way based on robots.txt and the flux monitor, at last it is proved to be an effective strategy and method to protest the Web site though experiments.