Automatic identification of text genres and their roles in subject-based categorization

Genre characterizes text differently than the usual subject or prepositional content that has been the focus of most information retrieval and classification research. We developed a new method for automatic genre classification that is based on statistically selected features obtained from both subject-classified and genre-classified training data. The main idea of the genre classification method is to calculate the weight of a feature for a genre class by using its frequency statistics for subject classifications. Having observed that the deviation formula and discrimination formula using document frequency ratios work as expected, we went on to study the roles of various types of features such as content-bearing words, function words, morphemes, and punctuations marks. In the first part of this paper, we present some of our findings in the roles of the feature types for genre classification, with a brief discussion of the genre-based classification. The genre classes we used are those often found in Web documents: accident reportages, newspaper editorials, personal homepages, product reviews, product specifications, research articles, and Q&A´s. The second part of the paper addresses the issue of how text genres help classifying documents based on the subject content of documents. This is a corollary to our original hypothesis that subject classification would help identifying the genre class of a document automatically. Our experimental work shows that while subject classes clearly help improving the genre-based classification, it is not clear whether using the genre class information for documents in the same way helps subject-based classification. However, we found that training a subject classifier with a set of documents belonging to a particular genre class improves subject-based classification.