Towards modernised and Web-specific stoplists for Web document analysis

Research areas such as text classification and document clustering underpin many issues in Web intelligence. A fundamental tool in document clustering is a list of ´stop´ words (stoplist) that is used to identify frequent words that are unlikely to assist in classification and is hence removed during pre-processing. Current stoplists are outdated both in light of fluctuations in word usage, and innocent of ´Web-specific´ stop words, hence questioning their applicability in Web-based tasks. We explore this by developing new word-entropy based stoplists: one derived from random Web pages, and one derived from the BankSearch dataset. We evaluate these against other stoplists using accuracies obtained from unsupervised clustering experiments. We find that existing stoplists perform well, but are sometimes outperformed by our new stoplists, especially on hard classification tasks.