Iterative Denoising using Jensen-Renyi Divergences with an Application to Unsupervised Document Categorization

Iterative denoising trees were used by Karakos et al. (2005) for unsupervised hierarchical clustering. The tree construction involves projecting the data onto low-dimensional spaces, as a means of smoothing their empirical distributions, as well as splitting each node based on an information-theoretic maximization objective. In this paper, we improve upon the work of (Karakos et al., 2005) in two ways: (i) the amount of computation spent searching for a good projection at each node now adapts to the intrinsic dimensionality of the data observed at that node; (ii) the objective at each node is to find a split which maximizes a generalized form of mutual information, the Jensen-Renyi divergence; this is followed by an iterative Naive Bayes classification. The single parameter α of the Jensen-Renyi divergence is chosen based on the "strapping" methodology, which learns a meta-classifier on a related task. Compared with the sequential information bottleneck method, our procedure produces state-of-the-art results on an unsupervised categorization task of documents from the "20 Newsgroups" dataset.