Mining contents in Web page using cosine similarity

Web pages typically contain a large amount of information that is not part of the main contents of the pages, e.g.; banner ads, navigation bars, copy right and privacy notices, advertisements which are not related to the main content (relevant information). In this paper, an algorithm is proposed that extract the main content from the web documents. The algorithm based on Content Structure Tree (CST). Firstly, the proposed system use HTML Parser to construct DOM (Document Object Model) tree from which construct Content Structure Tree (CST) which can easily separate the main content blocks from the other blocks. The proposed system then introduce cosine similarity measure to evaluate which parts of the CST tree represent the less important and which parts represent the more important of the page. The proposed system can define the ranking of the documents using similarity values and also extracts the top ranked documents as more relevant to the query.