An In-Depth Cross-Layer Experimental Study of Transport Protocols over Circuits

Recently, both research-and-education networks (RENs) and commercial networks have added a dynamic circuit service. With this service, users can request and obtain dedicated bandwidth for short durations (on the order of minutes to hours). Host-to-host (cluster-to-cluster) circuits are used for large file transfers. The network nodes used for this circuit service are not pure TDM or WDM circuit switches; instead they are hybrid nodes that include Ethernet interfaces that allow for the connection of host Ethernet NICs. While as with ordinary circuit switches, cross-connections have to be provisioned prior to data transfer, the technology allows for the creation of mismatched rate circuits, in which the wide-area portion of the circuit can be matched to the (typically lower) disk-access rates, while the access links from hosts to these switches are Gb/s Ethernet. In this work, an in-depth experimental investigation is performed to gain insights into the complex interactions between the TCP layer, ON/OFF flow control at the Ethernet layer, and switch buffer sizes (the Ethernet line cards have buffers). Using a novel tool and transport protocol designed for circuits, these interactions are characterized. An interesting new dynamic was found between flow control at the data-link layer and congestion control at the transport layer. With this in-depth characterization, it is clear that automated mechanisms are necessary to not only configure end-host TCP send and receive buffer sizes as is required for high throughput across IP-routed networks, but for circuit networks, additionally, Ethernet-layer output queue (called qdisc in Linux) size needs to be set, along with flow-control related parameters within switches.