Efficient topologies for large-scale cluster networks

Increasing integrated-circuit pin bandwidth has motivated a corresponding increase in the degree or radix of interconnection networks and their routers. This paper describes the flattened butterfly, a cost-efficient topology for high-radix networks. On benign (load-balanced) traffic, the flattened butterfly approaches the cost/performance of a butterfly network and has roughly half the cost of a comparable performance Clos network. The advantage over the Clos is achieved by eliminating redundant hops when they are not needed for load balance. On adversarial traffic, the flattened butterfly matches the cost/performance of a folded-Clos network and provides an order of magnitude better performance than a conventional butterfly. In this case, global adaptive routing is used to switch the flattened butterfly from minimal to non-minimal routing - using redundant hops only when they are needed. Different routing algorithms are evaluated on the flattened butterfly and compared against alternative topologies. We also provide a detailed cost model for an interconnection network and compare the cost of the flattened butterfly to alternative topologies to show the cost advantages of the flattened butterfly.