A measurement-based study of big-data movement

Parallel TCP connections are used for large scientific dataset transfers to increase throughput. Therefore, to accurately characterize big-data movement, it is important to reconstruct parallel flowsets from traffic measurements. In this work, we start with NetFlow records collected in an operational research-and-education network across which large scientific datasets are moved routinely, reconstruct individual elephant flows from the NetFlow records, and assemble parallel flowsets from elephant flows. Our findings are as follows. The top 1% of flowset sizes were in the hundreds of GBs to low TBs range, 95% of flowsets had rates less than 2.5 Gbps, and 99% of flowsets had durations shorter than 4 hours. Median flowset rate increases and rate variance decreases with increasing number of per-flowset component flows. Such findings are useful for network planning, traffic engineering, and for improving user performance, since large dataset transfers are among the most demanding of network applications.