Algorithm of web page purification based on improved DOM and statistical learning

In order to effectively remove the noisy information existed in web pages, such as advertisement, not related links, etc, and to improve the classification results, we proposed the algorithm of web page purification based on improved DOM tree and statistical learning. In this paper, we firstly establish block tree model by combining DOM tree and visual characteristics of web content, then statistical learning methods are used to discriminate each sub-block tree to identify the main content of the theme-based web pages. Experiment shows that the method has a good purifying effect for all kinds of theme-based web pages, the method can be applied to preprocessing stage of web page classification, which will enhance the accuracy of classification.