A supervised approach to distinguish between keywords and stopwords using probability distribution functions

This paper presents a novel probability based approach for distinguishing between keyword and stopword from a text corpus. This has a lot of applications including automatic construction of stopword list. First objective of this paper is to investigate the role of probability distribution for distinguishing between keyword and stopword. Second objective is to compare the performance of probability distributions of various weighting measures for the purpose of identifying keyword and stopword. Main characteristics of our method are that it is corpus base, supervised and computationally very efficient. Being corpus based the method is independent of the language used. However we have tested the approach on a domain specific corpus in Hindi. In Hindi (including many Indian languages), it has a great significance as a standard list of stopwords is not available. The results are encouraging and we are able to achieve 74% accuracy. However as this is a preliminary attempt, there is a great scope for improvement.