Social Streams Blog Crawler

Weblogs, and other forms of social media, differ from traditional Web content in many ways. One of the most important differences is the highly temporal nature of the content. Applications that leverage social media content must, to be effective, have access to this data with minimal publication/acquisition latency. An effective Weblog crawler should satisfy the following requirements: low latency, highly scalable, high data quality and appropriate network politeness. In this paper, we outline the Weblog crawler implemented in the social streams project and summarize the challenges faced during development.