NetFlow-based network traffic monitoring

In order to achieve real-time traffic monitoring on high-speed backbone link, this paper proposed a series of effective solutions aiming at data collection, processing and statistical analysis with minimal packet loss rate (even none packet loss). First of all, with NetFlow sampled records as data source, the proposed method effectively improves data collection efficiency by using buffer structures and multi-threads concurrent mechanism. Secondly, on the purpose of avoiding redundant operations in latter analysis to improve efficiency of the whole system, the method introduces a common operational procedure to make a unified process on raw data. Finally, through geographical region partition of the network and appropriate time granularity, the method analyzes network traffic performance between managed network (or a single IP) and outer networks. Using an ordinary PC, the scheme could be able to collect, process and analyze data from 20 Gbps backbone network with a high performance (packet loss rate less than 10^-6, memory overhead of 38 MB).