Enhancing Entropy-based Informative Block Identification Using Block Preclustering Technology

Identifying informative blocks to extract valuable content from web pages is a typical but crucial task in the web mining field. Currently entropy-based informative block extraction approaches achieve both high precision and recall rates. However, they are unable to identify blocks containing a few terms that are used frequently in the main text. To overcome this drawback, we propose a novel approach, called block analyzer, which preclusters blocks based on their structure. An entropy value is then assigned to each cluster as its weight, which is used to determine whether the blocks in the cluster are informative or not. Our experiment results show that about 70% of blocks collected from five types of web site were classified as either noisy or informative by both our method and an entropy-based approach. While the other 30% of blocks were judged as informative by both human analysis and our method, but not by the entropy-based method.