Design and Implementation of a Multi-Area Distributed Crawler Based on Skipnet-YL Network

Refering to single area crawler model based on p2p-chord network, designed and implemented a multi-area crawler model based on own design Skipnet-YL network which based on Skipnet protocol. It use better performance router protocol Skipnet-YL and the idea of SuperNodes and OSPF. Sequentially, it can solve the problem of low performance and the large of communication traffic during increasing nodes. After testing and analysing this model, this paper found out its deficiency, then put forward improved solutions for shortage. Eventually, it looked forward into future work.