Refining LDA Results and Ranking Topics in Order of Quantity and Quality with an Application to Twitter Streaming Data

Topic model is an emerging approach to summarize data, especially text data, in terms of a small set of latent variables. The most useful implement of topic model is LDA method, which is an unsupervised machine learning technique to identify latent topic information from a massive document collection. However, sometimes the LDA method gives some hard understanding or meaningless results. In order to improve this problem, in this paper we proposed a method for refining results of LDA and also ranking topics in order of some significance criterion. Our study is based on two basic assumptions. The first assumption is that the correlation coefficient between any two different topics should be zero under ideal condition. The second assumption is that the quality of topics can be defined as a deviation from background topic. Starting from these two assumptions, we provided a concrete method to determine the number of topics when using LDA method to extract topics from documents data and also to ranking the LDA results in order of quality. As a confirmation of our proposed methods, we conducted several experiments to processing Twitter streaming data. The results of these experiments show that our methods work efficiently as expected.