Profile-Based Focused Crawler for Social Media-Sharing Websites

In this paper, we present a novel profile based focused crawling system for dealing with increasingly popular social media-sharing Web sites. In this system, we treat users´ profiles as ranking criteria for guiding the crawling process. Furthermore, we divide a user´s profile into two parts, an internal part, which comes from the user´s own contribution, and an external part, which comes from the user´s social contacts. In order to efficiently and effectively extract data from a social media-sharing website for focused crawling, a path string based page-classification method was first developed for identifying list pages, detail pages and profile pages.