Research and implementation of the technology supporting MicroBlog data collection based on web crawler

MicroBlog is an effective vehicle for the network public opinion, and plays an important role in dissemination of the public opinion. A crawler which consisted of user crawling and contents crawling used to crawl MicroBlog is designed. The crawler used protocol-driven strategy, event-driven strategy and template extraction methods to achieve the successful extraction and data storage. Experiment shows that the crawler has an efficiency and integrity of information collection compared with the crawler BFS. A more flexible crawler is needed with the more complexity of DOM Tree.