Classification of unknown mobile web traffic based on correlation coefficient measurement

Classification of mobile web traffic on dimensions of domain names is critically important for Internet Service Provider (ISP) to understand the situation of mobile Internet, so as to balance the network resource allocation. In this paper, we propose a novel method as the first step on this topic, to identify unknown IP addresses, by using statistical information derived from massive data collected from a commercial ISP covering an entire city in southern China. Specifically, we create a Jaccard coefficient measure method to extract a domain name which is most relevant to an unknown IP from textual HTTP accessing logs. Results and analysis show that our method can deal with over 50% HTTP traffic of mobile network and achieve a relatively high accuracy to 85% and recall to 50% Furthermore, we design a set of parallel algorithms and implemented them based on MapReduce framework, which is a distributed parallel programming model with proved low-cost and high-efficiency features.