A Generalized Links and Text Properties Based Forum Crawler

Web forums have become a major source of information gathering/mining due to a large amount of user generated content. Crawling of Web forums is necessary to gather/mine the information from them. However, a generic Web crawler is unable to efficiently and effectively crawl the Web forums because of the existence of many redundant and duplicate pages. In addition, there exists a crawling relationship among the useful pages that need to be considered. So, for efficient crawling, we need to intelligently crawl the Web forums by eliminating redundant and duplicate pages, and understanding the crawling relationship. Existing works in forum crawling use visual pattern recognition based methods, which make them extremely computational expensive. In this paper, we propose a novel light-weight crawling method using text and links properties of the pages in Web forums. Theoretical analysis and experimental results show the effectiveness and efficiency of the proposed method.