A distributed multi-tasking job scheduling mechanism for web crawlers

Recently, the prosperity of social network nourished web services such as virtual community and web community. With the readily available social networking sites and the accessible internet, the interaction between people has become much frequently than before. Therefore, this research aims to provide assist to social networking sites by collecting and analyzing the enormous data on these sites. Currently there are many scholars putting effort on the research of data collection. We focus on the difficulties and challenges which web crawlers will encounterduring data collection and improve the mechanism proposed in [6]. In this paper, we will look deep in to the blocking of social networking sites and the derivative problem of idle web crawlers.