A Novel Method to Extract Informative Blocks from Web Pages

This paper proposes a novel algorithm to extract the informative blocks from web pages and filter the advertisement which has noting to do with the subject when people browse the Web page. In this pager, we use HTML parser to construct DOM tree and apply corresponding rules to construct a new tree (CST) which can easily help us to separate the ldquoprimary content blocksrdquo from the other blocks. Then we will use our algorithm to analysis CST and trim off useless blocks which are on the CST. The algorithms can identify primary content blocks by looking for the blocks that contains much more numbers of the block content. Our system can extract web content which is existed as the Table format or the Div format well. At last, Experiments on a set of more than thousands of web pages from 5 different sites show that the method is practical, and can achieve high accuracy.