A Wordsets based document clustering algorithm for large datasets

Document clustering is an important tool for applications such as search engines and document browsers. It enables the user to have a good overall view of the information contained in the documents. The well-known methods of document clustering, however, do not really address the special problems of text document clustering: very high dimensionality of the document, very large size of the datasets and understandability of the cluster description. Also there is a strong need of hierarchical document clustering where clustered documents can be browsed according to the increasing specificity of topics. Frequent Itemset Hierarchical Clustering (FIHC) is a novel data mining algorithm for hierarchical grouping of text documents. The approach does not give reliable clustering results when the number of frequent sets of terms is large. In this paper we propose WDC (Wordsets-based Clustering), an efficient clustering algorithm based closed words sets. WDC uses a hierarchical approach to cluster text documents having common words. WDC found scalable, effective and efficient when compared with existing clustering algorithms like K-means and its variants.