Optimizing collective communications on SMP clusters

We describe a generic programming model to design collective communications on SMP clusters. The programming model utilizes shared memory for collective communications and overlapping inter-node/intra-node communications, both of which are normally platform specific approaches. Several collective communications are designed based on this model and tested on three SMP clusters of different configurations. The results show that the developed collective communications can, with proper tuning, provide significant performance improvements over existing generic implementations. For example, when broadcasting an 8 MB message our implementations outperform the vendor´s MPl_Bcast by 35% on an IBM SP system, 51% on a G4 cluster, and 63% on an Intel cluster, the latter two using MPICH´s MPl_Bcast. With all-gather operations using 8 MB messages, our implementation outperform the vendor´s MPI_Allgather by 75% on the IBM SP, 60% on the Intel cluster, and 48% on the G4 cluster.