Collecting, managing and analyzing social networking data effectively

Social network analysis is a hot topic in recent years. However, how to collect, manage and analyze those large volume data effectively, is a very important problem. In this paper, we describe a system to collect, manage and mine social media data. Taking Tencent microblogs as an example, we implement a social media data collecting module based on Hadoop. In order to process data effectively and efficiently, we design a storm based managing and preprocessing system. As to the social media data analysis, we conduct topic detection based on LDA algorithm. We also propose a method to calculate the hot degree for each topic. In order to analyze the topic dynamics, we present a topic tracking method. Furthermore, we give an integrated method, which combines emotion and dictionary to analyze the sentiment of each topic. The Web-based result visualization are also implemented.