A novel approach to netflow monitoring in data center networks

Scalable netflow monitoring is one of the challenges in data center networks which have thousands of Virtual Machines (VMs) running on hundreds of physical servers. The typical architecture for netflow monitoring using a centralized netflow collector is not scalable in data center networks. The solution has been to use a cluster of netflow collectors and include a load balancing element in the network. In this paper, we propose a different architecture wherein the netflow clients directly write to a scalable NoSQL database such as Cassandra. We modified Open vSwitch (OVS) to directly write netflow packets to Cassandra. We extended ntop-ng, a free open source netflow collector software, to store netflow records in Cassandra and compared its performance with that of OVS, a netflow client, writing to Cassandra directly. We found that using Cassandra from OVS leads to a significant performance improvement over ntop-ng in terms of the average number of netflow packets written per second. We modified ntop-ng to store the netflow statistics to Cassandra and compared the performance of Cassandra against RRD, which is the default non-volatile storage in ntop-ng. We found that Cassandra is more scalable than RRD as the time to store the records increases linearly with number of writes for RRD whereas Cassandra has constant time in our experimental setup.