Title :
Hardware- and software-based collective communication on the Quadrics network
Author :
Petrini, Fabrizio ; Coll, Salvador ; Frachtenberg, Eitan ; Hoisie, Adolfy
Author_Institution :
Div. of Comput. & Computational Sci., Los Alamos Nat. Lab., NM, USA
Abstract :
The efficient implementation of collective communication patterns in a parallel machine is a challenging design effort, that requires the solution of many problems. In this paper we present an in-depth description of how the Quadrics network supports both hardware- and software-based collectives. We describe the main features of the two building blocks of this network, a network interface that can perform zero-copy user-level communication and a wormhole routing switch. We also focus our attention on the routing and flow control algorithms, deadlock avoidance and on how the processing nodes are integrated in a global, virtual shared memory. Experimental results conducted on 64-node AlphaServer cluster indicate that the time to complete the hardware-based barrier synchronization on the whole network is as low as 6 μs, with very good scalability. Good latency and scalability are also achieved with the software-based synchronization, which takes about 15 μs. With the broadcast, similar performance is achieved by the hardware- and software-based implementations, which can deliver messages of up to 256 bytes in 13 μs and can get a sustained asymptotic bandwidth of 288 Mbytes/sec on all the nodes. The hardware-based barrier is almost insensitive to the network congestion, with 93% of the synchronizations taking less than 20 μs when the network is flooded with a background traffic of unicast messages. On the other hand, the software-based implementation suffers from a significant performance degradation. With high load the hardware broadcast maintains a reasonably good latency, delivering messages up to 2KB in 200 μs, while the software broadcast suffers from slightly higher latencies inherited from the synchronization mechanism. Both broadcast algorithms experience a significative performance degradation of the sustained bandwidth with large messages
Keywords :
computer networks; concurrency control; multicast communication; performance evaluation; AlphaServer cluster; QsNET; Quadrics network; collective communication patterns; deadlock avoidance; latency; multicasts; network design; parallel machine; routing; scalability; Bandwidth; Broadcasting; Communication switching; Degradation; Delay; Network interfaces; Parallel machines; Routing; Scalability; Switches;
Conference_Titel :
Network Computing and Applications, 2001. NCA 2001. IEEE International Symposium on
Conference_Location :
Cambridge, MA
Print_ISBN :
0-7695-1432-4
DOI :
10.1109/NCA.2001.962513